We are in Washington D.C. this week exhibiting at the FOSE show and we're showing this awesome demo that we've been working on. Pervasive has taken Pervasive DataRush to new levels in order to meet specific cyber security challenges. It’s no secret that we’ve entered the big data era. There are millions of devices generating data every second: log events, security events, network traffic, firewalls, and so much more (this various is shown as shapes in the diagram below). And there’s lots of great software out there to look at these events, but only for a short time frame. One of the most daunting challenges facing organizations today is capturing, archiving, and analyzing ALL this data at any given time. There’s so much data that today’s software is failing to archive and analyze cyber security events as a WHOLE.
Until now.
Pervasive has developed a Historical Event Processing proof of concept (POC) that leverages and exposes the power of Pervasive DataRush as it captures and archives one million events per second into Hadoop’s HBase. Holy smokes, that’s amazing! This consumption rate is orders of magnitude faster than any solution on the market today.
For this POC, we used a single server box with 48 cores, 40 drives, and 258G of memory. But the processing rate will increase with increased cores or multiple nodes. We used Pervasive DataRush listeners for multiple log events and archivers to write to any database. And we actually captured 1.6 million events in 62 seconds, to be exact. Once the millions of events are captured and archived, you can use Pervasive DataRush to launch any set of queries or apply data mining algorithms to perform deep analytics on the dataset as a whole to look for a change in patterns...particularly useful in cyber security. For this POC, we used a Hive query to count server process types that generated each event and calculate percentages. Once the query was completed, Pervasive DataRush sent counts of each message to Google Charting to display the data visually. The entire process helps meet the demands of today's cyber security challenges, but we're especially impressed about the speed that we're able to capture this events and run queries.
If we captured a million events in one minute, imagine how much data is created everyday. Now organizations can capture all the data, archive it, and perform deep analytics as a whole. Stop by our booth #1528 at FOSE to see the POC in person.
