Taking it to the Extreme: Computing and Analytics on the Edge

In the world of data processing, speed might not be everything, but it sure is close. That is especially true when it comes to sensitive data such as found within national security, law enforcement and public safety.

To help clients maximize the ability to analyze huge amounts of data, Jacobs introduced Extreme Search, a suite of AI-empowered analytics functions that provide insanely fast statistical reasoning, predictive analytics and query capabilities that can perform in a wide variety of digital environments.

Recently, we sat down for an interview with one of the key proponents behind this new technology – Doug Wolfe, vice president and general manager within Jacobs’ Divergent Solutions operating unit. Doug helped explain the challenges that Extreme Search was designed to overcome, some of its key benefits and how this technological wonder from Jacobs is a differentiator for the clients who depend upon us.

Hi Doug. Can you talk about the problem or challenge that was being solved by the rollout of Extreme Search?

The core challenge it solves is the ability to go through really large amounts of data (either data being generated in real time or else a massive pile of data you may have been handed), and then search, filter and find data of interest. That allows you to use that thin dataset to feed into some of the more advanced algorithms and analytics, as opposed to going through a large “extract, transform and load” (ETL) and indexing process upfront, which pretty much all the big data tools do.

Typically, the search speeds are good once you have everything indexed, but to do so you often must pay a huge price upfront, and it takes a long time. Extreme Search provides speed – it allows you to define what you're looking for and then it provides a lot more focus on the data that you're finding and possibly forwarding to another analytic tool. The idea was to be able to go through the data raw and look at the patterns in the data and the pattern-matching, and to be able to digitally do those matches, as opposed to doing a database ingest first and then doing all indexing across each one of those data types. With Extreme Search, you can very quickly go through and find user ID or country code or whatever you might be looking for and focus on results in that data and perform further analytics if needed.

When it comes to fast data processing needs like we’ve been talking about, why do our clients pick us? What is our differentiator?

The main reason is that with Extreme Search, you can go through large data sets a lot quicker, and then it's a lot less expensive if you don't have to run all the compute and the database storage to ETL and index all of the data. There are real efficiencies up front and if you look at the data generators, they're far outpacing the IT tools' ability to ingest that data. In the big data environment, we've got sensors, cameras and microphones and other data generators. There is a lot of digital data on the web, but the generation of that content is growing at a rate that is greater than what you can ingest into databases. That problem just gets worse every day. That's the problem that we are trying to solve with Extreme Search.

What do you think is the most pressing problem for organizations in terms of the cybersecurity angle with data processing speed?

The most pressing cyber issue which we address with Extreme Search is the amount of data generated from your network, data on what's happening in the network and how it's performing. That's a huge amount of raw data. If you try to do the traditional database process with all that raw data, you just fall behind quickly. Most people select subsets of the data, and that’s a reasonable approach given the limitations of the existing tools. But if you're trying to go through six months of logs and answer the question, "Have I ever seen this malware or event in my dataset before?", you're in a difficult spot if you have to use traditional tools to do that.

What do you see are some of the top benefits that Extreme Search provides to clients?

One benefit is you have some confidence that you've gone through all your data at least with a basic filter. Frequently, a lot of data is just left behind or not analyzed because it's viewed as not needing to be analyzed, but if you can go through all of it, then you have a chance of taking at least a quick look at the data.

The timeliness is super important because you can go through a massive amount of data at search speeds that can keep up with the incoming data. There are use cases where customers need to make real-time decisions based on selected indicators, and Extreme Search enables faster decision-making. I think the other big benefit is that it fits nicely in the ecosystem where if you've got a thinned dataset (let's say by country code or network ID,) and if you rapidly send a much smaller dataset into an analytic tool, that analytic tool's going to perform much, much faster than if you feed the entire data volume into that tool and begin indexing.

It's a way to really get your arms around a huge amount of data volume in the most efficient way. We have seen at least an order of magnitude of performance improvement, especially when you avoid that upfront indexing step. It all depends on the data volume, data type, and just what kind of search you're trying to do. There are a lot of caveats around that, but an order of magnitude improvement is something we can achieve – 10x speed improvement to getting to answers in a large data volume.

Can you speak to the potential that edge computing advancements, such as Extreme Search, can help with sustainability?

We have not measured actual power savings, but there are energy savings by going through data at the source, and then filtering or thinning data and only forwarding on the network and into the cloud or data center what's relevant. I think the bigger opportunity is to locally optimize the functions close to the operations such as in water treatment or in a satellite operation, analyze those functions and optimize consumption based on algorithms that can be run with a small edge compute device.