The foundation of any intelligence-driven mission is built on the intelligence cycle—a six-step process used by government and defense entities to turn raw information into actionable intelligence.
Whether the goal is to address cybersecurity risks, combat terrorism, support law enforcement, or inform other national security initiatives, the intelligence gathering process involves:
- Identifying information requirements, mission objectives, and other factors to inform the cycle’s planning and direction.
- Collecting raw intelligence from a variety of relevant sources.
- Processing that data to return only what’s useful, and transforming it into appropriate formats.
- Analyzing and contextualizing data into finished intelligence.
- Disseminating and communicating that intelligence effectively.
- Incorporating feedback into the initial planning and direction phase.
There is no silver bullet solution for this process. Military intelligence requires a varied toolkit to meet requirements at each stage of the intelligence cycle.
Echosec Systems focuses on delivering solutions that address gaps in data collection, processing, and intelligence analysis. How do Echosec Systems products fit into the intelligence cycle at these stages, and what does this look like in a practical defense application like counter-terrorism?
Intelligence professionals can identify the data types and sources required once a mission’s intelligence requirements, planning, and direction are clearly defined.
While government and defense often require human, signals, electronic, and imagery intelligence, Echosec Systems focuses on providing open source intelligence (OSINT) from a variety of online networks. This includes:
- Deep and dark web data, including marketplaces, chan sites, paste sites, forums, and chat applications
- Publicly-available data from a number of widely used and more obscure social media and news sites
- Geospatial data from social media
Data diversity is key in the intelligence cycle—yet there are a number of critical OSINT sources that are not offered through commercial, off-the-shelf APIs and other data discovery solutions. These include less-regulated social networks like Gab, Telegram, and 4chan, as well as paste sites useful for detecting data leaks.
Echosec Systems specializes in offering comprehensive access to a number of these niche sources. These networks are increasingly valuable for defense applications, particularly those centred on counter-terrorism as more extremist communities migrate to less-regulated social platforms.
An intelligence analyst was gathering data for a counter-terror objective and required Telegram, Gab, and 4chan data for monitoring threat actor plans, recruitment, and other communications in a specific extremist group.
The Echosec Systems Platform API enabled the analyst to collect relevant data from these sources through the intelligence agency’s existing UI, which wasn’t possible through other commercial vendors. The analyst was then able to search and view specific extremist activity without needing to create an account, navigate the sites manually, or request membership to any closed Telegram groups.
The intelligence cycle is ineffective without streamlined processing, even with access to relevant data sources. This is where collected data is filtered down to only what’s relevant for mission requirements, transformed into a usable format, and integrated into other data feeds and tooling. Lower-level analysts also need to view and process data in an intuitive UI so they can get up and running quickly.
Echosec Systems enables defense users to accomplish this in a few different ways. Raw data collected through Echosec Systems products are indexed, normalized, and in some cases, tagged with machine learning classifiers for specific threat categories like data disclosure and identity hate.
This allows defense analysts to efficiently reduce noise and extract or search for high-priority entities including author identifiers, crawled and published dates, personally identifiable information, and board names. Echosec Systems Platform tools are also simple and user-friendly, allowing for fast and easy querying and results navigation. Results are automatically sorted by relevance using the TF-IDF model.
Data collected through the Echosec Systems Platform API can also be easily integrated into bespoke solutions and external intelligence products, which are typically required by data scientists in defense. When data can be processed against other feeds, analysts can glean powerful insights more efficiently as they move into analysis.
A data scientist working on the same counter-terror objective needed to integrate Telegram, Gab, and 4chan data into external tooling. With the Echosec Systems Platform API, they integrated social data from these feeds, allowing end-users to collect, filter, and immediately cross-reference it against other data points and OSINT feeds in their toolkit. This allowed the analyst to reduce false positives and link an extremist Telegram user to an account on another social site as well as a documented real-life persona.
As discussed above, integrating and cross-referencing disparate data feeds is also helpful for improving and accelerating data analysis. Processing and analysis steps in the intelligence cycle are fluid, often overlapping—particularly when using technologies like machine learning to contextualize data.
According to the United States Intelligence National Strategy (2019), the intelligence community is increasingly challenged by growing volumes of online data for collection, processing, analysis, and triage. This is exacerbated by a data analyst shortage. These challenges point to a growing need for advanced machine learning applications to automate intelligence cycle processes like analysis, which tend to rely heavily on HUMINT.
Echosec Systems helps automate data analysis with proprietary machine learning models. This allows users to find more relevant results faster for a number of common threat categories. Data crawled by Echosec Systems is also stored as a data lake, which allows data scientists in defense to access a rich repository of catalogued historical data. This is valuable for developing machine learning models and other automations specific to defense.
The analyst submitted an API query to the Echosec Systems “Social” endpoint for a number of keywords unique to an extremist group they were monitoring online. Since they have access to over 1 million unique data points through the Echosec Systems Platform API each day, they focused only on posts ranked highly under the “Threat” category. This category automatically detects content indicating intent for pain, injury, or violence toward an individual or group. The analyst discovered a number of immediate, actionable threats faster and reduced time spent contextualizing data.
The data scientist was also able to build new machine learning models from a data lake using social data crawled by Echosec Systems. These models were built to detect and rank content indicating an active threat, such as a mass shooting, by this particular extremist group.
The intelligence cycle is an ubiquitous model in the intelligence community. However, as use cases evolve, OSINT sources expand, and technologies like machine learning improve, intelligence professionals are forced to reevaluate this approach for each mission: Where are there source gaps in intelligence collection? How can data feeds be processed and integrated to get more value from them? And how can automated analysis more accurately and efficiently support HUMINT?
Tools along each stage of the intelligence cycle must become more specialized and advanced to address these questions and satisfy mission requirements in the public sector.
Need to integrate niche social data to satisfy intelligence requirements? Contact us to learn more about our API.