The UK’s public sector urgently needs OSINT technology, writes Dr Brenton Cooper, CEO and co-founder, Fivecast, which offers open-source intelligence (OSINT) software.
Open-source data has grown to become immensely valuable in policing, defence and national security, but the scale of information available and the complexity of analysing it for intelligence purposes is now beyond human capabilities.
For the UK public sector, which, like many governments around the world, is under significant financial pressure, any expansion of capabilities through recruitment of highly paid data scientists is unlikely. Spending on policing in England and Wales, for example, is expected to take a five per cent real-terms hit into 2028-2029, according to the Institute for Government.
Yet the need for greater efficiency in the UK’s OSINT (open-source intelligence) gathering and analysis has never been more urgent. OSINT capabilities are increasingly relied on by police forces and organisations like the National Crime Agency, HMRC, the Serious Fraud Office and the Financial Conduct Authority. OSINT complements the insights organisations gain from many other forms of intelligence including human or classified intelligence.
These are capabilities that are increasingly prized by US and NATO armed forces. The US army released its OSINT strategy last year, for example, prioritising the use of open-source data along with AI and advanced analytics. And 11 NATO nations are participating in the Northern Raven initiative to improve OSINT collection and sharing in relation to Russia, part of a wider cultural change in attitudes to open-source intelligence backed by the US.
In the policing sector, forces know criminals use the dark web to collaborate illicitly, while activists leave a trail on many public platforms as they plan activities that have potential to cause major disruption or disorder.
Collecting the right data from these sources and analysing it to detect suspicious connections, trends and threats across the web, social media channels and the dark web takes a great deal of time and expertise when conducted manually. Statista calculates the volume of online data will reach 181 zettabytes by next year. This volume of multi-media data and its complexity make the task too onerous for under-pressure human analysts without advanced OSINT tools.
Many police forces, however, still only have embryonic OSINT capabilities. In an article he wrote for the Police Foundation in 2021 before becoming Metropolitan Police Commissioner, Sir Mark Rowley described OSINT as a Cinderella service, requiring a cultural shift to ensure it fulfilled its full potential. That was prior to the current financial situation.
UK Security Vetting, set up in 2017 to vet people in the public sector, is a good case in point of the need to invest in OSINT. Last year the National Audit Office found the agency was falling behind on developed vetting, clearing only seven per cent of cases within 95 days, against a target of 85 per cent. Its “continued poor performance” risked government departments being unable to proceed with work relating to national security. The service was still using a system that demanded many manual workarounds.
In May this year, the CEO of UKSV reported the service was hitting its targets after an “arduous” recovery plan that included digital streamlining of processes. In the current security and geo-political environment, it is likely, however, that much greater automation will be required to handle the growth in vetting demand and the need to analyse huge volumes of data far more quickly.
AI and machine learning are essential
For vetting as much as for detection of organised crime, the effective exploitation of OSINT demands AI, machine learning and allied analytical technologies such as natural language processing. Piecemeal implementation will not work. Organisations need solutions that dovetail with their own workflows and operate intuitively, vastly augmenting the expertise of their experienced intelligence and security professionals. This is the only way to deliver the full riches of OSINT so organisations can transform their mission-critical decision-making.
In 2022, a joint paper between the think tank RUSI and the Alan Turing Institute recognised this, urging greater use of OSINT for national security purposes. As well as the exponential growth of data, the paper pointed to a “favourable cost equation” making public data less costly to analyse. It also highlighted the increasing importance of using publicly available and covertly collected intelligence in tandem. OSINT technology will free up advanced intelligence gathering talent for “high-end” work and fill a gap in tradecraft, the authors said.
They were correct. The implementation of OSINT platforms that use AI and machine learning to complement human analytical skills, stands to transform so many aspects of policing, national security and vetting. Advanced, highly automated platforms can rapidly identify high risk content, criminal networks and relationships that would otherwise take days or weeks manually.
The reach of such platforms extends right across billions of data points, a huge range of forums, platforms and official information sources, and the dark web. Smart prioritisation cuts out the noise while augmented intelligence enhances but does not replace human decision-making and improves the processing of data.
This is a far more advanced set of capabilities than the use of spreadsheets or crude search capabilities deployed for a few hours a week. Users can customise the AI-driven risk detectors to detect novel risks quickly and efficiently across large datasets, but without depending on access to specialised data science expertise. Sentiment analysis, rapid filtering and detection of key words and phrases, along with built-in obfuscation to ensure anonymity, are all part of the armoury.
The design and deployment of AI must be guided by fundamental principles that ensure it remains within ethical constraints and avoids bias and harm, is reliable, safe, accountable, transparent and explainable.
The automation capabilities of such OSINT platforms supercharge intelligence teams and the process of verifying information and establishing digital footprints across multiple accounts, public records, news sources, the dark web and many more publicly and commercially available data sources. Agencies save thousands of hours in what would otherwise be time-intensive and repetitive tasks, increasing the level of accuracy.
The ROI and security gains from adoption of advanced OSINT technology are now very clear-cut when the UK’s public finances are under such intense pressure and the volume of threats and risks to society are i constantly increasing. All of these factors are driving the increased and urgent need for OSINT as a key piece of the intelligence puzzle that can help deliver greater public safety, reduced risk and vastly improved investigation efficiency at lower cost in what may be troubled times.
