6 Reasons Why Open-Source Intelligence is Climbing the Priority Ladder
In 2021, the CSIS Technology and Intelligence Task Force upgraded OSINT to “core” intelligence and called for a new intelligence agency—the Open Source Intelligence Agency—as an option to address OSINT’s growing value.
“The task force has concluded that the IC must fundamentally
reconceptualize OSINT as a cornerstone of U.S. intelligence,
relevant across the IC enterprise and in all aspects of its
current and future missions. Unlike the other ‘INTs,’
however, OSINT does not have a dedicated agency.”
In the last few years, we’ve seen a global pandemic kill millions, social media target free societies with disinformation, and nation-states invest in rapid military and technological advancements. OSINT now provides the information needed to keep governments more informed in global information environments and protect populations against security risks.
Where does this OSINT come from, and what do OSINT applications look like in a modern operations center?
Emerging Open Source Intelligence Sources
Open-source intelligence is gathered from publicly available sources. This includes offline open sources like public documents and in-person interviews but is now primarily comprised of online sources. So what’s out there?
The surface web includes any content that is indexed and searchable with free search engines like Google. The Intelligence Community can gather OSINT like indexed social media posts, maps, and public records from the surface web.
Social media content is usually associated with social networks like Facebook, photo and video-sharing sites like YouTube, interactive media like TikTok, and microblogging sites like Twitter. However, OSINT is also discoverable on more covert social sources, including:
- Decentralized networks (e.g. Mastodon)
- Alt-tech sites (e.g. Parler)
- Messaging apps (e.g. Telegram)
- Forums and imageboards (e.g. 4chan)
Even though these sources may be harder to access, have a smaller user base, or host different content types than mainstream social media, they make up an important part of the social mediaverse—especially for OSINT analysts.
Regional Information Sources
The IC needs to stay informed about global information environments. Location-specific social media, news sites, and forums help intelligence professionals investigate regions of interest in more depth, often from on-the-ground sources. For example, social networks like Naver (South Korea) or Sina Weibo (China) are valuable OSINT sources for generating intelligence on these geographic areas.
The deep web makes up an estimated 90% of internet content and includes any unindexed web pages (undiscoverable through standard search engines). Many of the social media and regional sources mentioned above are on the deep web, and it also hosts non-social sites relevant for open-source intelligence, like paste sites.
The dark web makes up a small part of the deep web and is only accessible through the Tor browser, which anonymizes user communications. Dark web content gives intelligence professionals a window into illegal activities like trafficking, fraud, and hacking. However, it also hosts open-source intelligence sources similar to those on the surface and deep web: social networks, paste sites, forums, messaging services, and news aggregators.
Modern Open-Source Intelligence Applications for the IC
1. Malign Influence Monitoring
Jeffrey Rosen, Deputy Attorney General at the US Department of Justice, describes malign influence as foreign preferences that occur through coercive, covert, or corrupt activities. The goal of these activities is typically to influence democratic processes like elections, or, in Rosen’s words, “to influence policymaking and public discourse more broadly.”
This could look like disinformation or media manipulation. Illegal measures, like hacking confidential emails, can also be used for malign influence. In the last decade, widespread social media use has enabled malign influence operations to reach larger audiences and target them more strategically with algorithms and automation.
Media manipulation can look like:
- Impersonating professional or personal social media accounts
- Spreading false or misleading information about a government, individual, or event
- Creating photos or videos that do not represent reality
- Reposting false content on legitimate sources
- Misleading phrases or hashtags that quickly gain popularity
Monitoring public social data helps the IC better track and understand disinformation campaigns and other forms of malign influence on the web.
For example, Russia has been accused of media manipulation tactics in the ongoing conflict with Ukraine. Gathering OSINT from news and social media sources relevant in that region helps intelligence professionals understand emerging narratives and their reach as the conflict unfolds.
2. Crisis Response
The COVID-19 pandemic has demonstrated the value of OSINT during a crisis, from contact tracing efforts to supply chain monitoring. Online spaces are often the earliest public information sources to inform an effective response.
For example, social media users often post public updates and images from the scene of a crisis. Aligning this data with other intelligence sources can help provide a faster, more informed response.
Whether it’s a natural disaster, public health crisis, or terrorist attack, open-source intelligence can help intelligence teams answer:
- Where is the crisis, what does the impact look like, and what will happen next?
- Where are resources required?
- How are other countries responding and how can that inform our strategy?
- What information do the public, first responders, and other public sector entities need?
- How is the crisis being co-opted by adversaries, like disinformation actors
3. Counter-Terrorism and Extremism
For years, online spaces have been used by global terrorists to push extremist propaganda, recruit, fund, and organize. In 2021, this activity also became recognized as a domestic tactic after the January 6th insurrection on Capitol Hill—which was organized openly on social media networks like Parler.
Open-source intelligence supports counter-terrorism efforts by illuminating how extremist groups of any background operate. This can help inform decisions to support individuals who may be vulnerable to recruitment. The IC can use OSINT to understand and combat emerging social media sources and content used to spread propaganda and recruit. OSINT can also help counter-terrorism efforts to slow or block terrorist financing, which often relies on online channels.
We’ve also seen in domestic attacks that social media is valuable for early warning indicators or leakage, which can help predict public safety risks like shootings.
Between the work-from-home movement, the Solarwinds hack, and the rise of ransomware, cybersecurity has become top-of-mind for governments.
Breaching government data is financially and politically lucrative for lone-wolf attackers, organized hacking groups, and nation-state actors. Cyber capabilities are becoming more sophisticated and accessible to adversaries. Persistent online threats include:
- Breaches and cyber espionage. Adversaries, especially nation-state, target public sector data to assess technology advancements and leverage other strategies like media manipulation. Critical infrastructure systems are also financially lucrative for ransomware attacks.
- Network attacks and take-downs. Distributed denial of service (DDoS) attacks can take down critical government systems and are commonly perpetrated by hacktivist groups and nation-states.
- Botnets. These can infect computer networks with malware or support media manipulation efforts like disinformation.
Cyber attacks also increase in response to events that elevate public anxiety, like COVID-19. For example, there was an increase in malicious domains providing health information, government and public health entity impersonation, phishing, and ransomware attacks early in the pandemic.
OSINT sources like paste sites, forums, and marketplaces reveal early breach indicators and emerging attack techniques. Combining this information with cybersecurity feeds helps the IC more confidently predict, mitigate, and investigate cyber risks.
5. Geopolitical Assessments
Open-source intelligence supports intelligence efforts to understand global information environments and situational awareness more generally. Social media, news, and forums—especially those used in target regions—are valuable for assessing:
- New technological advancements, like weapon systems and AI, developed or deployed by other nations. This OSINT data can surface on unindexed forums—for example, Arab Defense Forum or Pakistan Defence.
- Public sentiment or unrest in an area of interest.
- Media manipulation tactics in areas of interest.
- On-the-ground activities that indicate geopolitical tensions or conflict in areas of interest, such as a buildup of military infrastructure.
This information can be used alongside classified intelligence sources, like signals intelligence (SIGINT) or technical intelligence (TECHINT).
6. Foreign Liaison
OSINT resources and access also help governments improve intelligence partnerships with foreign states. According to CSIS, this could extend beyond Five-Eyes nations:
“An expanded OSINT mission is particularly well suited to building intelligence partnerships with foreign liaisons. Instead of just sharing intelligence, U.S. OSINT analysts could collaborate with foreign counterparts in building assessments from the ground up. Alongside analysts, data scientists and AI experts could partner to test, train, and develop algorithms and applications.
In addition to the Five-Eyes (FVEY) intelligence alliance, OSINT liaison partnerships with innovative services with common security threats such as Israel, South Korea, Japan, and the Nordic/Baltic states are ripe for expansion.”
A Note on OSINT Solutions: Addressing Intelligence Requirements
We’ve covered where relevant open-source intelligence originates and how it can be applied—but how can tools address modern IC requirements?
Any analyst knows that finding relevant OSINT is not viable using standard search engines. Intelligence professionals require specialized software to address requirements at each stage of the intelligence cycle.
Commercial open-source intelligence tools help teams gather data efficiently and in line with mission requirements. But because intelligence teams often work with proprietary tooling, they also require data integrations that plug into their existing systems.
According to the US Intelligence National Strategy (2019), the intelligence community is increasingly challenged by growing volumes of online data available for collection, processing, analysis, and triage. The western world is also facing a data analyst shortage.
Data accessibility is also a concern as the web evolves. For example, more covert sources (like alt-tech platforms, regional networks, and unindexed forums) do not offer their own API or are unavailable through commercial data providers. To gather data from these sources, analysts often need to create dummy accounts, make group requests, and navigate networks manually. This requires time and resources that could be allocated to other areas of the intelligence cycle.
To address these challenges and satisfy intelligence requirements, OSINT solutions must:
- Improve data coverage, providing access to emerging sources that are not commonly available through commercial, off-the-shelf vendors.
- Leverage machine learning capabilities. AI is a major priority for the public sector, and it can help analysts process and contextualize data more efficiently.
- Be intuitive and user-friendly for lower-level intelligence analysts, providing more efficient workflows and faster speed-to-information.
There’s no doubt that OSINT is becoming more valuable for the IC. The question is how is this information applied, which sources are relevant, and how can OSINT tools address IC requirements?
Classified intelligence sources are no longer enough to keep intelligence professionals informed when it comes to national security risks and general situational awareness. The proliferation of social media—including both mainstream networks and covert channels—now offers intelligence teams the information required to protect digital and physical assets, assess global information environments, and foster stronger intelligence partnerships.
Going forward, governments must prioritize OSINT solutions that address modern intelligence challenges. This means improving open source data coverage, usability, and AI capabilities. These features will enable more efficient and comprehensive intelligence cycles as OSINT becomes integral for a variety of intelligence applications.
Are you looking to expand data coverage in your intelligence toolkit?
Consult with us to make sure your bases are covered.