Enterprise Trend Visualization and the Potential of OSINT

Is it possible to build a dashboard and command center to give a 360-degree view of the emerging trends, topics and conversations affecting your enterprise? KPMG believes so. They have invested in Bottlenose, a cloud-based trend intelligence solution that analyzes real-time streaming data from social media and other sources. How much data? A Twitter feed is around a billion messages a day. Bottlenose is now analyzing a fire hose of over 72 billion records streamed daily, including 100% of Twitter and Facebook. The company has also found a way to include 98% of live TV and radio, by employing real time speech-to-text translation.

The Bottlenose solution is able to find patterns in big data streams from multiple sources, and cross-correlate them to meet the needs of a specific business stakeholder. A variety of algorithms are employed, drawn from the fields of sentiment analysis,  Natural Language Processing (NLP) and machine learning (ML). Results are presented to end users via dashboards and rich interactive visualizations. KMPG have cited potential use cases including risk and threat management, forensic investigations, cyber attack detection, crisis management, live event monitoring, competitive intelligence and customer growth analytics.

Solutions such as Bottlenose have far outgrown their roots in social media monitoring and corporate brand protection.

Quid is another startup operating in a similar space, although it should be stressed that there is no clear definition yet of what constitutes a “trend visualisation” solution.


Quid integrates data from more than 250,000 sources. It is used by analysts, strategists and researchers to explore very specific topics and to help define strategies of relevance to their enterprise goals. For example, a hedge fund owner may use it to help develop a complex investment strategy. A market analyst or innovation practitioner may use it to prepare a proposal for new product development. Quid’s interactive visualizations help these intelligent users to explore millions of data points, revealing hidden links and features buried in the sea of information. They are able to pivot, filter and report on entities such as the names of companies, people, products, brands, financial instruments and specific technologies.

With no clear definition of either the scope or functionality of such solutions, we can expect startups operating in this space to focus on specific industry needs. Solutions offering only a generic capability may need to develop a strong vertical market focus if they are to have a viable business.

RecordedFuture is another fascinating company exploring the potential of big data. The heart of their offering is a patented temporal analytics engine. Think Google with added time machine! Here’s the idea: As well as extracting named entities from open web data, RecordedFuture finds the associated temporal references. Examples include fuzzy phrases such as “next month” and “by the end of next year” as well as precise dates found in the headers of structured data such press releases and news items. The solution is therefore able to present its results on timelines, representing how a trend is playing out or is expected to play out. It’s almost like being able to search the future!

recordedfutureCurrently collecting over 600,000 web sources in seven languages, and processing more than 5 billion Web changes to cast a wide net, RecordedFuture can be a powerful tool in the hands of a trained researcher. It has found a market need in strategic defence and enterprise security assessment, but has also been used for competitive product and market intelligence.

Open source intelligence (OSINT) is one term that is used to refer to insights collected and derived from publicly available sources, such as the web (or structured databases accessible via web APIs). Another related term is public open data.

The myriad data sources that solutions such as Bottlenose, Quid and RecordedFuture draw upon and can holistically visualise include:

  • Web media including news, magazines, and now radio and television streams
  • User-generated content from online communities, including social media, wikis and blogs
  • Public data from government departments, including budgets, contract awards, demographics, environmental data, public debates and press conferences
  • Professional and academic sources, including legal databases, subject-oriented conferences, expert journals and research papers
  • Geospatial information, including maps, boundary data, terrain data, urban data, marine data, transport movements (air, sea and land), and data derived from sources such as Google Earth, Maps and Street Views

Web formats such as HTML and XML – together with domain-specific metadata standards, and the availability of search engines and Web APIs – have greatly expanded the scope and range of the OSINT available to solution providers. In addition, providers often offer mechanisms for orienting a solution to a specific use case and for integrating open data sources with private enterprise databases – for example, biomedical research.

The boundary between applications and solutions in the space of OSINT and public data can be confusing.

Explorys, a healthcare big data solutions provider, has anonymized, and then integrated, more than 315 billion clinical data points over 50 million unique cared-for lives. This includes clinical, operational, and financial data elements, that when combined into analytical models, provide the insights that healthcare professional leaders need in order to make data-driven decisions about their care-delivery models.

explorysAs with the other solutions explored in this article, Explorys places great emphasis on the development of formalised metadata models (ontology) and the associated data acquisition and curation methods within their data grid architecture and data lifecycle processes. The approach adopted by Explorys has now touched more than 360 hospitals and is now serving the needs of more than 300,000 healthcare providers. Every time a new healthcare organisation or trust comes onto their platform, Explorys enjoys the advantages of a virtuous circle: access to new data sources, extending and enriching their evolving solution.

It is claimed that Explorys now provides the industry’s largest clinical and operational dataset, searchable in a privacy-protected and HIPAA-compliant manner. This capability enables inquisitive leaders within healthcare contexts to test hypotheses across statistically significant human populations, to ensure that their management decisions are sound.

We expect powerful data-fusion solutions to emerge in every industry niche and in every area of public policy, scientific endeavor and research.

marineexploreMarineExplore is another example. It provides a cloud-based platform where professionals and those with an interest in the marine environment explore, discover, donate and share public ocean data.  The site now has more than 7,000 professional users, has curated nearly 2 billion point measures from 40,000 in-situ platforms and 50 gridded data sources. The solution supports the community in many ways, including online collaboration, visual data exploration, development of specialised maps, dashboards and plots and in writing new data monitoring tools. Registered users can build custom data sets for their purposes in minutes from the integrated sources which include those from NASA, NOAA and other agencies.

The examples highlighted in this article demonstrate that the potential for vertical industry and OSINT solutions is only likely to increase over the coming years. We live in a time of wonder.

All images the property of the respective companies:



  1. This is because fraudsters goal the waste-disposal process specifically regarding
    files comprising data similar to this. Think of how much money you can conserve should
    you no-longer needed to pay for things like mouthwash, hair items, cosmetics, shavers and shaving

  2. Usually I don’t learn post on blogs, but I wish to say that this write-up very compelled me to
    try and do so! Your writing style has been surprised me.

    Thank you, very great post.

  3. Theere is definately a great deal to find out about this topic.
    I love all the points you made.

Speak Your Mind


This site uses Akismet to reduce spam. Learn how your comment data is processed.