Application programming interfaces (APIs) are becoming an integral part of any organization’s investment in digital transformation—and intelligence and corporate security entities are no exception.
APIs help connect data with applications, saving users the resources otherwise required to integrate data inputs manually. In the context of gathering threat intelligence, the quality of an API’s data and delivery is high-priority for:
- Defense and intelligence teams requiring access to online data feeds
- Corporate security operations centres using online feeds for security alerts
- Data companies seeking valuable online data inputs to offer their own clients
APIs are a relatively emerging technology. Over half of API developers have spent less than five years in the field, according to a 2020 report by SmartBear. For threat intelligence applications, this leaves room for improvement for commercial data and API providers—particularly when it comes to web data coverage.
Where do commercial API solutions fall short, and what do improvements mean for threat intelligence professionals working in the public and private sector?
Where Commercial APIs Fall Short
Diversifying access to online data is crucial as the digital world expands. This means analyzing both dark web and technical data, as well as less conventional feeds like unregulated social platforms and other obscure surface, deep, and dark websites. These alternative web spaces are becoming more relevant for threat intelligence feeds as adversaries explore new attack surfaces and use a variety of emerging platforms.
For security and intelligence teams, increased data coverage means less risks are overlooked as the online landscape evolves. Users also benefit by diversifying access to data suitable for training machine learning models, which helps support more efficient data analysis amidst a serious skills shortage.
The problem is that many commercial, off-the-shelf APIs do not combine conventional threat intelligence feeds with the more obscure online sources gaining relevance for corporate security and defense use cases.
For example, many Telegram groups, decentralized platforms, and imageboards are not directly accessible through most commercial API services—even though they are valuable for monitoring a variety of threat intelligence, from cybersecurity risks to terrorist activity.
For end-users who require access to these types of feeds, this can result in:
- Spending more time and resources manually navigating web spaces
- Creating dummy accounts to access semi-closed groups or platforms
- Missing out on posts that may have been deleted shortly after publication
- Overlooking relevant but hidden sources that analysts may not be aware of
- Retrieving raw data that cannot be easily applied to data science tasks like machine learning development
Industry research also suggests that data coverage should be top-of-mind for any organization seeking threat intelligence feeds.
According to a study by the Delft University of Technology, commercial threat intelligence vendors appear to vary widely in their data coverage—overlapping 4% at most even when tracking the same specific threat groups. This raises concerns about how much relevant data is missed by intelligence professionals, even when using multiple vendors.
What a More Holistic Solution Means for You
As relevant threat intelligence migrates to a greater mixture of online spaces, security and intelligence professionals are likely to prioritize expanded data coverage in their tooling.
This can be achieved by leveraging API vendors who offer a wider variety of standard and alternative threat sources than is commonly available through commercial solutions. This looks like combining standard threat intelligence feeds like onion sites and social media with emerging imageboards, paste sites, chat applications, and decentralized platforms.
This has a number of benefits. For one, more data coverage = less overlooked threat intelligence. Access within an API allows analysts to spend less time gathering data manually. Crawling more obscure sources also means that any posts that have since been deleted from the original site are retained for analysis—a side benefit not available through manual collection.
Additionally, combining various inputs allows for easier cross-referencing and pivoting between data sources. This is valuable since the intelligence bread crumb is becoming more convoluted as the online risk landscape diversifies and expands. As a result, analysts can glean insights that might not be obvious or available when standard and alternative data feeds are not integrated.
A more data-diverse solution can also better support machine learning development. Without access through an API, many online sources—such as content on obscure social sites and chat applications—could not otherwise be catalogued and stored appropriately for data science applications.
The Echosec Systems API
At Echosec Systems, we recognized the unique value of combining standard and more obscure online sources for our clients, and built an API to address their data coverage requirements.
This is valuable for intelligence professionals, security operations centres, or other data companies who don’t necessarily require a pre-built Platform UI—just access to high-quality data from a range of sources. The API serves as an ideal complement to your existing threat intelligence feeds, such as technical data.
Right now, the Echosec Systems API provides access to:
- Dark web (e.g. Tor sites, I2P)
- Deep web (e.g. IRC, forums)
- Mainstream social media platforms (e.g. Reddit, YouTube)
- Obscure social sites, forums, imageboards (e.g. 4chan, 8kun)
- Messaging apps (e.g. Telegram, Discord)
- Decentralized platforms (e.g. Mastodon, Gab)
- Paste sites (e.g. Pastebin, DeepPaste)
The API delivers uniquely-sourced data. For example, we’ve built proprietary web crawlers that sit in applications (e.g. Telegram chat rooms) to gather data not available through other API offerings.
This is the only commercial API that combines uniquely sourced, obscure data feeds with the more mainstream, conventional feeds that security and intelligence teams require.
A comprehensive threat intelligence solution can save organizations millions, uphold national security, and retain public trust in the target organization. As online platforms evolve, critical information can be easily overlooked if security operations centres and intelligence agencies aren’t looking beyond standard sources.
As a wider array of online spaces become relevant to security initiatives—whether it’s a private or public sector environment—addressing data requirements in the coming years will rely heavily on the breadth of sources available through commercial API solutions.
Are your analysts manually navigating obscure sites for threat intelligence? Contact us to see how we can help.