Pervasive DataRush

Going really fast ...

jfalgout — Thu, 11 Mar 2010 22:05:00 GMT

In a recent blog, Robin Bloor discusses how parallelism is required for software to go really fast on todays multicore computers. He brings up this point about MapReduce: using MapReduce on problems it wasn't intended to solve is "... like playing golf with a single club". I'd like to expound a bit on this analysis.

MapReduce was most famously implemented by Google to fulfill their need to index the world wide web. Quite an undertaking! And MapReduce proved to be critical to their success. The programming model for MapReduce fits perfectly with the problem of finding words within documents and creating indices for later (very fast) lookup.

However, the MapReduce programming model can be limited when applied to other, more complex problems. Many deep data analysis algorithms require multiple, complex steps to produce their output. In these cases, a more general use programming paradigm is a better and more efficient fit. Hence, Robin's analogy to "... playing golf with a single club".

Robin goes on to discuss DataRush and the capabilities it brings to bear. Based on a dataflow architecture, the programming model of DataRush is much more flexible and general use than MapReduce. I wouldn't use DataRush to index all the content of the internet, but it has proven to be an excellent tool for general data processing and data mining. And it has the ability to utilize all of the cores available on today's multicore systems. Put into perspective, we have benchmarks showing Terabyte an hour (and even better) processing of network log data for a cyber security application on a single box.

So does playing golf with only one club mean you can't play golf? Of course not. But it does mean you can't play as well or as efficiently as if you used all the clubs at your disposal. The flexibility of DataRush allows you to utilize a full programming paradigm especially suited to big data problems.

Robin has a knack for the turn of a phrase. Check him out on Twitter. He also has a very funny (and informative) book out called "Words You Don't Know". I especially like the chapter on swear words.

HIMSS 2010 - DataRush in Health IT making a difference in patient care

n5712036 — Thu, 11 Mar 2010 14:13:00 GMT

The DataRush team just returned from attending the HIMSS 2010 conference in Atlanta (March 1-4^th).

HIMSS is traditionally a large conference. As massive as the Atlanta convention center is, this year once again HIMSS filled the venues. Total attendance on March 2nd was recorded at 27,451 with 13,530 as exhibitors. This years’ theme was “Health IT (HIT) making a difference in patient care”. The DataRush team staked its camp at booth#3325 where it visited with presenters, vendors and attendees discussing large data pains and synergies between interoperable health IT and the DataRush parallelism engine.

Day 1: Started early with a lunch briefing with a global leader IT services. The architect of the largest healthcare informatics data warehouse shared his experiences with content management and searchable health information. Performance bottlenecks are common in data warehousing (DW) and BI data loading and consumption. Additionally, consolidation of data from up to 40 member companies had its challenges with speed and accuracy of data de-duplication. In reference to DW loading, Pervasive DataRush offers a platform that leverages multi-core technology enabling unprecedented data throughputs. In response to data quality and consolidation, Pervasive DataMatcher combines the power of DataRush parallelism engine with sophisticated parametric string matching algorithms to provide the highest precision (fidelity) and recall (completeness) in fuzzy matching.

Once the exhibitor’s area opened midmorning, DataRush booth featured live demos of Pervasive DataRush Analytics plug-ins for KNIME. KNIME (Konstanz Information Miner) is an open source data mining tool. The upcoming DataRush release 5.0 provides KNIME nodes that offer DataRush powered versions of several data mining algorithms. The KNIME interface provides a user friendly workflow-like environment for data miner practitioners to rapidly orchestrate and deploy predictive analysis. Healthcare system integrators delivering EMR and EHR based decision support are turning to predictive analytics and data mining. Pervasive DataRush speeds and accuracy combined with the KNIME interface enables instant processing and response from Tera-scale heterogeneous data repositories.

Day 2: Booked back-to-back with briefings to target specialized areas in Healthcare data exchange, interoperability, network support and cyber security. Healthcare data exchange deals with transformations between End-User formats and ANSI EDI, HL7, etc. standards. These transformations are facilitated via ETL tools. Pervasive DataRush powered schema-to-schema transformations supports healthcare standards as defined by the American National Standards Institute (ANSI).

Within cyber-security, Pervasive DataRush presented results from a benchmark study based on Malstone algorithm for web exploits. The Malstone DataRush implementation is capable of processing 1Tb of web log files per hr. Furthermore, DataRush is 12 times faster on 19 less computers when compared to the Malstone algorithm implementation using MapReduce on a 20 node cluster. Customers requiring enhanced cyber security discussed their bottlenecks in vulnerability assessments. An important task in cyber security involves analyzing large volumes of network data against databases of Common Vulnerabilities and Exposures (CVE) software flaws, Common Configuration Enumeration (CCE) mis-configurations, etc. and then scoring them based on National Institute of Standards and Technology (NIST) severity scoring system for vulnerability. The Common Vulnerability Scoring System (CVSS) provides an open framework for communicating the characteristics and impacts of IT vulnerabilities. DataRush offers data parallelism where subsets of network data can be concurrently read in blocks and processed. The processing here refers to concurrent computation of CVSS core equations for “AccessComplexity”, “Authentication”, “AccessVector”, “ConfidentialityImpact” and “AvailabilityImpact”. The final step in the processing pipeline is to aggregate these scores to produce the CVSS “BaseScore”.

Regarding network support and interoperability, discussions were centered on DataRush implementations of parallelized encryption and decryption algorithms as well as baseline change detection algorithms to detect anomalies in network log files. These are novel applications of DataRush driven entirely by customer need.

Day 3: Had the most traffic through our booth. Fraud, waste and abuse in healthcare came up frequently among vendors and attendees. The Pervasive DataRush team featured live demo of the Benford’s Law Fraud Detection operator recently added to the Pervasive DataRush-Analytics library and the DataRush KNIME plug-ins package. Implementation and application of this operator is explained in detail in my blog titled “Finding a needle in a hay stack”, click here to read.

Other topics of interest at HIMSS included Image Processing and Image Content Management. Image management solutions today use picture archiving and communication systems (PACS). PACS is a combination of hardware and software dedicated to the short and long term storage, retrieval, management, distribution, and presentation of images. There are a lot of inefficiencies in image processing pipelines that could benefit from DataRush parallelism. Higher quality, efficient packaging and faster delivery of images results in earlier diagnosis and characterization of disease with the potential to improve patient outcomes and reduce costs.

To summarize, we have found healthcare IT to be a great fit for DataRush parallelism engine and its analytics operator library. HIMSS was a successful event for Pervasive DataRush Team in the number of leads generated, the networking opportunities and the flow of ideas in the application of DataRush to operations and analytics in healthcare.

Cluster in a box

jfalgout — Fri, 05 Mar 2010 14:29:00 GMT

Processors continue to become more "core dense" as we approach a watershed mark in multicore development. We've approached the point where a 4 processor system is taking on the characteristics of a "cluster in a box". Mike Hoskins, our CTO at Pervasive has deemed March 2010 the "Multicore Month of March" - a clever alliteration alluding to the latest processor announcements of Intel and AMD. Intel is launching their Nehalem EX with 8 cores and AMD their Magny Cours processors with up to 12 cores. This AMD blog outlines a contest to put an AMD 48-core system to good use. The winner to receive a free 48-core box.

But what to do with all those cores? At Pervasive, we've been developing a platform called DataRush that helps developers of all skill levels write high performing, scalable Java code. Based on a dataflow architecture, the underlying framework has a very simple programming paradigm. Using a shared-nothing approach, the developer is freed from having to manage hundreds of threads, worry about synchronization and deadlock or deal with other parallel programming headaches.

We recently came across a set of cluster-based benchmarks called MalStone. The MalStone Benchmark was developed by the Open Cloud Consortium (www.opencloudconsortium.com). For additional information, see www.opencloudconsortium.org/benchmarks. Even though these benchmarks are targeted at clusters, we decided to build a DataRush application for the MalStone B benchmark and run it on a 24 core system. The results were astounding!

The MalStone B benchmark consists of a 10 billion row log file containing site-entity records. The file size approaches 1 Terabyte in size. The purpose of the benchmark is to compute a ratio for each site (w), per week (d), for all entities that visited the site, the percent of visits for which the entity became compromised at any time between the fisit and the end of week d. This type of processing is typical for cyber security applications.

For our testing at Pervasive, we used a single machine consisting of 4 AMD 8435 processors running at 2.6 GHZ. This gives the machine a total of 24 cores. The total memory capacity is 64 Gigabytes. The I/O system uses a RAID filesystem consisting of 5 SATA drives. This machine does have a large memory capacity, however during the test we generally utilize about 9 Gigabytes for the JVM.

Using this system we were able to achieve a 68 minute runtime for the MalStone B benchmark. That's approaching a Terabyte an hour of processing on a single node, inexpensive server-class machine. Contrast this runtime to the same test run on a 20-node cluster of 4-core boxes using Hadoop. The MalStone B runtime for the Hadoop cluster is 840 minutes. DataRush on a single node machine is 12 times faster than the small cluster! To be fair, the processers in the cluster nodes are not exactly comparable to the ones in the DataRush test box. This test still shows that an inexpensive, single server-class machine with software like DataRush that can take advantage of multicore processors can outperform distributed processing. Cluster in a box in action!

This is very exciting when put in context of the next generation of multicore processors due out in the "Multicore Month of March". We look forward to running this and other benchmarks on these new systems. Check back soon for updated numbers as I have a sneaking suspicion that we'll easily break through the Terabyte an hour threshold with the MalStone benchmark using the latest Intel and AMD systems.

PMML validation

wtian — Wed, 17 Feb 2010 17:06:00 GMT

Predictive Model Markup Language (PMML) is the leading standard for statistical and data mining models. PMML describes one or more structures of the data mining models in XML document with a root element of type PMML.

Our Pervasive DataRush-Analytics project provides the following data mining models: AssociationModel, NaiveBayesModel, and RegressionModel. The PMML generated from these models can be shared and exchanged from one environment to another, but the PMML needs to be validated against the schema to find any problems that may need to be fixed.

To guarantee validation, the Pervasive DataRush-Analytics model uses both XSD validation and XSLT validation as recommended by data mining group.

First step: XSD Validation :

Get the PMML XSD 3.2 schema

Here is an example of validating PMML file against PMML XSD schema:

public void pmmlXSDValidate(String schemaPath, String sourcePath) {

try {
SchemaFactory factory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);Source schemaFile = new StreamSource(new File(schemaPath));
Schema schema = factory.newSchema(schemaFile);

Validator validator = schema.newValidator();

validator.validate(new StreamSource(sourcePath));
} catch (SAXException e) {

..........
} catch (IOException e) {

..........
}

}

XSD validation is a necessary part, but not sufficient by itself for determining if a PMML model is valid.

Second step: XSLT Validation:

Get the PMML XSLT style sheet

Here is an example of XSLT validation.

public void pmmlXSLTvalidate(String stylesheetPath, String sourcePath, String resultPath) {

try {

DocumentBuilderFactory docFactory = DocumentBuilderFactory.newInstance();

//This setting will ignore the namespace
docFactory.setNamespaceAware(false);
DocumentBuilder parser = docFactory.newDocumentBuilder();
Document document = parser.parse(new FileInputStream(sourcePath));
Source pmmlSource = new DOMSource(document);
Source xsltSource = new StreamSource(new FileInputStream(stylesheetPath));
TransformerFactory transFactory = TransformerFactory.newInstance();
Transformer transformer = transFactory.newTransformer(xsltSource);
transformer .transform(pmmlSource , new StreamResult(resultPath));

//check result after transformation

......................
} catch (TransformerConfigurationException e) {

..............
} catch (TransformerException e) {

..............

}

}

It is possible that problems may still exist even if the PMML is validated, but running this test lowers the probability. Once validated, Pervasive DataRush-Analytics models will provide specified results to help you analyze your business data and predict customer need.

DataRush Video Analytics

n5712036 — Wed, 17 Feb 2010 14:42:00 GMT

Digital video has become the face of television, the internet and mobile devices. According to an official blog post (May 2009), about 20 hours of video are introduced to the YouTube site every minute of real time. This is equivalent to Hollywood releasing over 114,000 new full-length movies into the theaters each week! But digital video also plays a huge role in biomedical devices, surveillance and manufacturing quality assurance.

Did you know there are approximately eight million users sharing 10 petabytes of data (mostly media files) at any given time? This accounts for nearly 10% of the worldwide internet broadband connections [1]. So how can near real-time actionable intelligence be gleaned from the vast amounts of video data being generated? One answer is to exploit the power of emerging commodity multicore computers. When used properly, each core can be used for individual threads of computations, but new software applications will need to be developed to make this happen. Today there is a parallel programming gap between multicore systems and software applications. With the end of the uni-processor performance gains, the average software developer will have to implement parallel programs to maintain performance growth. The goal in parallel computing is to perform multiple calculations simultaneously. The Pervasive DataRush™ (DataRush) platform exploits multiple forms of parallelism facilitating concurrency in video processing and video analytics from spatial-temporal partitioning and down to the pixel level.

There are several paths to parallelism given languages and programming frameworks available today, but a very common path to parallelism today is data parallelism. Data parallelism is a simple divide-and-conquer technique emerged from SPMD (single program, multiple data) where data is partitioned and distributed over multiple workers (nodes on a cluster, vm’s on a cloud) each running the same program. Hadoop, an open source version of MapReduce, logically partitions the data and allocates one map task, called a Mapper, per partition. There may be hundreds of Mappers on a single machine. A single threaded legacy application can be deployed to subsets of a large scale datasets on a cloud or grid environment. A second path is coarse grain parallelism via parallelization of loops (TPL: Parallel.For, Parallel.ForEach and RParallel: runParallel), arrays (ParallelArrays and INVOKE-IN-PARALLEL) and further orchestration onto multiple workers. True fine grain parallelism requires writing complex and correct multithreaded programs. Fine grain parallelism here refers to thread-level parallelism (not instruction level parallelism).

This figure is a cartoon depiction of a data pipeline for Video Object Detection using principal component analysis (PCA) for background subtraction. By projecting the original frame onto its eigenspace and subtracting projected image from original image, foreground objects are clearly identifiable. This work is based on Yilmaz et al (2006). A white paper detailing this work can be found here.

Video analytics can also be used in medicine for guided surgery and video tele-monitoring of patients. A use case (see Figure 2 below) and task in this video processing pipeline is the identification of regions of interests for physicians and clinicians decision support. DataRush parallelism has being applied in experimentations with K-Means clustering of digital colposcopic images to identify acetic acid enhanced pre-cancerous legions (highlighted in red below). This image analysis can be applied concurrently to individual video frames in order to identify and label ROI's.

The DataRush platform is designed specifically to fully utilize emerging commodity multicore computers. It addresses gaps in design time cost, programming, parallelism, scalability and performance/watt, enabling rapid prototyping of video processing applications.

The volumes of video streaming onto television, computers and mobile are forcing video processing onto cloud and distributed environments. Current cloud and grid computing platforms are still not capable of real time processing. Video processing is inherently parallel and the current solutions mostly leverage data parallelism. Emergent fine grain parallelism in video processing exploits concurrency at slice-level, frame-level, intra-frame and pixel-level operations. Such fine granularity has been traditionally achieved using video encoding hardware. This hardware based approach lacks flexibility. Our approach introduces a Video processing development platform to exploit multiple levels of parallelism while facilitating rapid development of agile and adaptive video analytical models.

What could you do with a 100x performance improvement?

azeemj — Tue, 09 Feb 2010 13:46:00 GMT

Traditionally Java Performance has always been a bit of a misnomer. In the early days of Java 1.1, performance was a secondary consideration after ease of programming and usability. But the last few years have seen some amazing performance enhancements in the JVM. Escape Analysis, Compressed References, and other JDK7 performance enhancements. I'm not even including the myriad of smaller changes that the Sun JVM engineers are working on.

As a Java developer, you don't have to wait for JVM improvements to get better performance out of your code. There are best practices on how to use the Java language (Java Concurrency, Java Performance Tuning) and specialized hardware such as Azul Systems that can help. But what if after doing all sorts of optimizations you're code STILL isn't running as fast as you want it? Especially with regards to using multiple cores, which can be difficult to fully utilize.

The DataRush team has been working on DataRush 5.0 which is scheduled to be released sometime in the middle of 2010. Until the release of the new version of DataRush, I thought that showing you some of the performance that we're seeing with the current builds. All of the applications were run on the following system config:

4p/24c AMD Opteron 8435 2.6GHz
64gb DDR2
Windows 2008 R2 64bit
Java6_u16 64bit Server JVM

The following algorithms were run:

The run time for each algorithm were:

Naive Bayes

Learner - 3.6 seconds
Predictor - 7.8 seconds

Kmeans - 3.2 seconds

The DataRush engine fully utilized all twenty four cores through the complete run! The Naive Bayes algorithm was run on an 8gb data file, while the Kmeans was run with a 3.2gb data file. There are more algorithms that the DataRush team is working on and these are the first set of impressive results.

What does this mean for your business? Imagine being able to do complex calculations, analytic and other algorithms in seconds! Instead of waiting to run calculations overnight, you could run computations as needed and as often as needed. Update models in near real time, taking different inputs and view results as they happen to make better business decisions based on your whole data set instead of a sample. If this sounds interesting, head over to Pervasive DataRush to get your two week trial of DataRush.

Pervasive Software Among 10 IT Companies to Watch in 2010

livey — Wed, 06 Jan 2010 16:35:00 GMT

Just days into 2010 and Robin Bloor includes Pervasive among ’10 IT Companies to Watch in 2010’ for the second year running. Pervasive lands second on his list and Bloor believes we will be “worth watching this year.”

Although Pervasive Software is comprised of four innovative technologies, Bloor praises Pervasive DataRush as the “so-far-little-recognized jewel” due to its parallel processing capabilities. Pervasive DataRush released the general availability of its processing engine in March, 2009. Bloor adds that “there are not many such engines and as we advance further into the world of multicore CPUs, every vendor that has a parallel engine is likely to experience strong demand. Parallel engines improve processing speeds by one or two orders of magnitude, bringing down query responses (for example) from hours to minutes.” Early adopters of the revolutionary Pervasive DataRush engine are taking advantage of the impressive speed ups and beginning to use DataRush to enable their own applications.

The Pervasive DataRush team worked diligently and built data-intensive applications useful in data services, data mining, and predictive analytics that were also released last year. Pervasive DataMatcher helps organizations detect fraud through duplicate records. Pervasive RushRecommender is a scalable, dataflow implementation of collaborative filtering based on weighted co-clustering that provides organizations insight to their customer needs.

Dr. Nena Marín, Pervasive DataRush Chief Scientist, co-presented with The University of Texas at the KDD Cup 2009 in Paris, France. The co-authored paper, titled “Pervasive Parallelism in Data Mining: Dataflow Solution to Co-clustering Large and Sparse Netflix Data,” was selected from among 686 total submissions and detailed work to deliver performance improvements in the Netflix recommender system running a computationally intensive co-clustering algorithm. The successful duo went on to present accurate prediction of customer behavior at Predictive Analytics World 2009 titled "Churn, Baby, Churn: Fast Scoring on Large Telecom Dataset."

Stay on the lookout as the Pervasive DataRush team uncovers new analytic applications in healthcare and cyber security, and new academic alliance advancements with TACC in the new year. Also on the roadmap is improved speed and content filtering. We continue to address the gap between proliferating multicore processors and exploding volumes of data to get you the information you need quickly. Please email us if you'd like to qualify for the Early Adopters Program.

Of teraflops and terabytes

kirwin — Wed, 06 Jan 2010 15:45:00 GMT

Before the holidays, I attended SC '09, this year's supercomputing conference. While supercomputing has traditionally been the domain of academia, the needs of business and scientific computing are converging. The datasets and processing needs of companies are growing such that the sizes of the problems addressed in both worlds are approaching the same. Hardware vendors are producing solutions which address both the need to store large volumes of data and to provide large amounts of computational power. But to do anything, the data and the computation must - at some point in time - be co-located. With conventional server computing, this isn't really a problem, since everything is local to the machine. In the more scalable architectures, however, data can be non-local. Furthermore, the cost of accessing remote data can be great (and even variable in grid-based architectures). So can you bring the terabytes and teraflops together to solve your problems?

The traditional supercomputing solution is message passing (MPI), moving the data to the computation. Nodes send messages to and receive messages from each other as needed to exchange data. A very flexible, low-level approach. But you also want to make sure you spend most of your time processing the data, not communicating with other nodes. I attended a number of presentations discussing aspects of this, such as: optimizing the I/O during the initialization phase, load balancing the workload, assigning work to nodes to minimize data transfer costs, utilizing asynchronous messaging to reduce stalls in processing.

Of course, the opposite approach is possible too, moving the computation to the data. In the ideal case, this what happens in map/reduce models like Hadoop. The computational and storage architecture is one; data is spread across the nodes, with each map running on local data. This means that distribution of the data becomes part of the storage - it doesn't completely disappear, but becomes a one-time, up-front cost.

And what about dataflow? As the name suggests, in the dataflow model the data moves. In fact, you can view dataflow as a more structured form of MPI. However, this structure provides one thing for free - pipelining allows computation and communication to overlap without special programmer effort. This works equally well for both disk I/O - you can get good throughput on sequential reads from disk - and for cross-node communication.

None of these are a universal solution. Each has strengths and weaknesses. As stated, MPI provides flexibility, but also places most of the responsibility on the programmer. On the other hand, while map/reduce and dataflow do much of this work transparently, they both require thinking about problems in a way which fit the paradigm - which may not be the intuitive approach to the solution.

Code coverage tool for Datarush

wtian — Mon, 21 Dec 2009 20:01:00 GMT

The Datarush project is built with Maven 2 and integrated with Hudson. Maven is a tool used for building and managing any Java-based project. Hudson provides automated, continuous build integration system, making it easier for developers to integrate changes to the project.

We need a good code coverage tool for the Datarush project. Code coverage is a measure used in software testing. It is a form of white box testing. Basic code coverage criteria include function coverage, statement coverage, branch coverage, condition coverage, etc.

EMMA is a free good java code coverage tool, but there is no Maven 2 plugin for EMMA now. EMMA’s current edition was publish in 2006 and hasn’t been developed for several years.

Sonar is another good code coverage tool. It requires a relational database for storage of measures data, but not able to keep report history. Sonar’s technical architecture includes cobertura , so it generates similar coverage reports as cobertura tool. The sonar web server requires large RAM to run efficiently.

We chose Cobertura as code coverage tool for our Datarush project. Cobertura is a free Java code coverage tool that calculates the percentage of code accessed by tests. It can be used to identify which parts of your Java program are lacking test coverage. Its coverage metric shows the code coverage percentage of packages, files, classes, methods, lines, and conditionals. Cobertura is based on jcoverage.

Cobertura can be used from Maven 2. Cobertura Maven Plugin provides the features of Cobertura within the Maven environment.

Example of basic configuration:

<project>
    . . .
    <build>
         <plugins>
                <plugin>
                     <groupId>org.codehaus.mojo</groupId>
                    <artifactId>cobertura-maven-plugin</artifactId>
                     <version>2.3</version>
                     <configuration>
                    <formats>
                         <format>html</format>
                         <format>xml</format>
                      </formats>

</configuration>

</plugin>

</plugins>

</build>

</project>

The report generated by this plugin is the result of executing the Cobertura tool against your compiled classes, and can be used to identify which parts of your java program are lacking test coverage.

Cobertura can be integrated with Hudson for building/testing software projects continuously.

Here is a sample cobertura code coverage report from Hudson:

Scalable Multivariate Linear Regression Enabled by DataRush

yangyou — Fri, 18 Dec 2009 20:24:00 GMT

Regression is a statistical term defined by Merriam-Webster's dictionary to be: "A functional relationship between two or more correlated variables that is often empirically determined from data and is used especially to predict values of one variable when given values of the others. "The regression value of y on x is linear"; specifically: a function that yields the mean value of a random variable under the condition that one or more independent variables have specified values."

Using DataRush, we implemented Multivariate Linear Regression using Ordinary Least Squares. Basically OLS "minimizes the sum of squared distances between the observed responses in a set of data, and the fitted responses from the regression model".

Trying to find coefficients (beta's) based on minimizing the sum of squares isn't straightforward or simple. However, it can be shown that (cf. Multiple Regression with a Singular Matrix, by M. J. R. Healy) the betas can be estimated by solving an upper triangular square matrix equation consisting of inner products between all the columns.

The DataRush implementation can handle virtually unlimited data sets and had good scalability on the number of cores, as evidenced by the following images, processing US census demographics dataset containing 63 columns. Compared to single threaded implementations, the speed up is significant (as much as 50x on a 24 core system).

Government Data Mining Opportunities Abound

Richard Maddox — Mon, 14 Dec 2009 21:17:00 GMT

Government data, when made available and presented effectively, can provide useful information to organizations and citizens alike. Star News Online presents compelling possibilities in the article, “Local Governments Offer Data to Miners.” From school violence to bicycle accidents to property values, public-sector data is increasingly available, wide-ranging and voluminous. Claire Cain Miller writes that government data in the hands of developers could result in applications to improve data usefulness, particularly for the public.

While there are some government organizations that do not offer public data to entrepreneurs, others are open to doing so if it can improve the lives of citizens. Even more, there is a growing consensus among public administrators to make data disclosure an official policy. While this is mostly taking place on the state and local levels, Miller relays that “the White House is about to publish a directive expected to give similar instructions to federal agencies.” A major point of this article, expressed by the executive director of an ecological nonprofit, says, “On detailed data dealing with very complicated material, you really have to know what you’re looking for in order to distinguish between good data and junk data.” Even more important, patterns and trends in data can only be unlocked with effective data mining.

With high-powered commodity multicore hardware now available, the right technology software platform can unlock patterns in even very large amounts of data at speeds unimaginable until now. Data Miners no longer need to rely on a sample set of large data, but can run terabytes+ in minutes on multicore servers. As a major player in the multicore revolution, Pervasive DataRush offers a rich, high-performance, automatically scalable architecture for data mining applications. You can also learn more about Pervasive DataRush in the public sector.

DataRush Line-By-Line: Hello, World!

jhefner — Thu, 10 Dec 2009 22:29:00 GMT

Let’s take a look at a basic "Hello, World!" DataRush application (click to download DataRush now!) and dissect it line-by-line. Below is the code we will be analyzing. See if you can get a feel for what is going on in the application; and, then scroll down to see how your guesses match up with what the application is actually doing.

==== SNIP ====
/*01*/ import com.pervasive.datarush.flows.RecordFlow; /*02*/ import com.pervasive.datarush.operators.ApplicationGraph; /*03*/ import com.pervasive.datarush.operators.DataflowNodeBase; /*04*/ import com.pervasive.datarush.operators.OperatorFactory; /*05*/ import com.pervasive.datarush.operators.io.textfile.ReadDelimitedText; /*06*/ import com.pervasive.datarush.operators.io.textfile.WriteDelimitedText; /*07*/ import com.pervasive.datarush.ports.RecordInput; /*08*/ import com.pervasive.datarush.ports.RecordOutput; /*09*/ import com.pervasive.datarush.ports.StringInputField; /*10*/ import com.pervasive.datarush.ports.StringOutputField; /*11*/ /*12*/ /*13*/ public class HelloWorld extends DataflowNodeBase { /*14*/ /*15*/ public static void main(String[] args) { /*16*/ ApplicationGraph app = OperatorFactory.newApplicationGraph(); /*17*/ RecordFlow names = app.add(new ReadDelimitedText("names.txt")).getOutput(); /*18*/ RecordFlow hellos = app.add(new HelloWorld(names)).getOutput(); /*19*/ app.add(new WriteDelimitedText(hellos, "hellos.txt")); /*20*/ app.run(); /*21*/ } /*22*/ /*23*/ /*24*/ private final RecordInput input; /*25*/ private final RecordOutput output; /*26*/ /*27*/ public HelloWorld(RecordFlow names) { /*28*/ this.input = newRecordInput(names); /*29*/ this.output = newRecordOutput(this.input.getType()); /*30*/ } /*31*/ /*32*/ @Override /*33*/ public void execute() { /*34*/ final StringInputField name = (StringInputField)this.input.getField(0); /*35*/ final StringOutputField hello = (StringOutputField)this.output.getField(0); /*36*/ /*37*/ while (this.input.stepNext()) { /*38*/ hello.set("Hello, " + name.asString() + "!"); /*39*/ this.output.push(); /*40*/ } /*41*/ /*42*/ this.output.pushEndOfData(); /*43*/ } /*44*/ /*45*/ public RecordFlow getOutput() { /*46*/ return this.output.getFlow(); /*47*/ } /*48*/ }==== SNIP ====

OK, have an idea about what’s going here? Let’s break things down.

Line 13: Our class is aptly named HelloWorld. We extend the DataflowNodeBase class, which is one of two primary base classes you’ll encounter when writing DataRush applications. The other base class is DataflowGraphBase. The difference between nodes and graphs is that nodes do work (they have an abstract "execute" method that must be overridden), while graphs hook nodes together (designating producer-consumer relationships).

Line 16: We begin our main method by creating an ApplicationGraph object using a factory method. ApplicationGraph serves as a container for the components of our application. ApplicationGraph also exposes a run method which invoke once all our components have been added to the container, in order to set all of those components in motion.

Line 17: We add our first component to our ApplicationGraph. It is a ReadDelimitedText component (included in the standard DataRush library) which reads a text file expressed in a field-delimited format. Its output is a stream of records that have been parsed row-by-row from the text file.

The add method on ApplicationGraph adds the new component to the container and then returns the component so we can invoke the component’s getOutput method. The RecordFlow object returned by getOutput() represents the stream of records.

Line 18: We add our second component to our ApplicationGraph. This is our custom HelloWorld component. We’ll be covering its functionality in subsequent lines; but, notice that we take the output from ReadDelimitedText—the "names" RecordFlow—and feed that to HelloWorld. Also, as with ReadDelimitedText, we invoke the new component’s getOutput method after add()-ing it to our ApplicationGraph. The RecordFlow that getOutput() returns represents the stream of records which contain our "Hello, <name>!" messages.

Line 19: We add our final component to our ApplicationGraph. This is a WriteDelimitedText component (also included in the standard DataRush library) which writes a stream of records to a text file in field-delimited format. It exposes no getOutput method because it serves as a data sink only. Notice that we take the output from HelloWorld—the "hellos" RecordFlow—and feed it to WriteDelimitedText to be written to disk.

Line 20: We invoke the run method of our ApplicationGraph. This method will cause all of the node components our container to do their work until all the data from our input file has been consumed. It’s important to note that the run method will not return until all nodes have finished executing.

Line 24: We start the real contents of our HelloWorld node component. Everything we’ve covered so far has been in our main method and has related to populating and running our ApplicationGraph. Something to keep in mind is that the placement of the main method in our HelloWorld class is arbitrary. We could have equivocally placed the main method in another class and our application would be unaffected.

Line 28: We create a RecordInput object for our node to pull its input from. In DataRush terminology, this RecordInput object is an input port. The input port must be backed by a stream of data, so we pass its factory method our "name" RecordFlow.

Line 29: We create a RecordOutput object for our node to push its output to. In DataRush terminology, this RecordOutput object is an output port. The output port must have schema or record type to base its outgoing stream of records on. In this case, we decide to use the same record type that our input port has and we pass that type to the output port factory method.

Line 33: We begin our node’s execute method. This method is overridden from DataflowNodeBase and will contain the core processing logic of the node. A node’s execute method is invoked only once during the application’s runtime and thus usually contains a loop for consuming or producing data.

Line 34: We extract from our record input port the specific field input port that yields names. Here, we get this field input port by index (it is the 0th field); but, if you know the name of the field from the schema or record type, you could also use that name.

Line 35: We extract from our record output port the specific field output port is used for pushing "Hello, <name>!" messages. Here, we get this field input port by index (it is the 0th field); but, if you know the name of the field from the schema or record type, you could also use that name.

Line 37: We start our core processing logic loop. The stopping condition for this loop is when we’ve consumed all the data on the stream of records that backs our record input port. All record input ports maintain a cursor on the stream of records that underlies them. Calling the stepNext method on an input port advances this cursor so that the next record in the stream can be accessed. When the cursor can no longer be advanced because end of data has been reached, stepNext() returns false.

Line 38: We do three important things in this line:
1. We read the current value of the "name" field input port as a String. Note that reading this value does NOT advance the cursor of our record input port.
2. We add a "Hello, " and "!" to the name that was read to form our "Hello, <name>!" message.
3. We set the current value of the "hello" field output port to our newly formed "Hello, <name>!" message. Note that setting this value does NOT actually push the value onto our output stream.

Line 39: Having set the value for our "hello" field output port, we push the value onto our output stream of records. The reason set() and push() are separate methods is that record output ports can contain multiple fields; and, each field is set individually before finally pushing the record. In this sense, field output ports act as registers for a record output port. Invoking a record output port’s push method then pushes the contents of its registers to the output stream.

Line 42: As the final action in our core processing logic, we call the pushEndOfData method on our record output port. This call is necessary to let nodes that are downstream know when to terminate their own core processing loop.

Line 46: Finally, we expose a getOutput method so that other components can consume our node’s output stream. Because components are designed to consume streams of data, rather than the record output port we have been using to push data, we return the RecordFlow yielded by our output port’s getFlow method.

Now that we’ve seen how our application works, let’s see it in action. Running the application with a names.txt input file like this:

==== SNIP ====
"Alice"
"Bob"
"Carol"
"World"
==== SNIP ====

Will yield a hellos.txt output file like this:

==== SNIP ====
"Hello, Alice!"
"Hello, Bob!"
"Hello, Carol!"
"Hello, World!"
==== SNIP ====

Note that the quotes in each file are actually field-start and field-end delimiters, and are a by-product of using the default configuration for ReadDelimitedText and WriteDelimitedText. Both components are designed by default to work with files expressed in comma-separated-value (CSV) format.

That's all for our line-by-line walkthrough! Copy and paste the code to try it yourself. I hope you learned something today; and if you have any questions or feedback please feel free to leave a comment.

Matisse for Eclipse?

rcauble — Tue, 01 Dec 2009 21:50:00 GMT

I've been working on KNIME plug-in development recently for Pervasive DataRush, and found myself wishing that there was a Matisse for Eclipse. As it turns out, as long as you are using Swing ( as is the case of KNIME ), you can use Netbeans to create a form which then runs in Eclipse! This is possible in the latest version of Matisse, since the generated java code uses GroupLayout, a standard part of the JDK.

The one trick is that Netbeans can only create a Matisse form in a Netbeans project. However, once created, it can be copied to an Eclipse project and then opened within Netbeans for additional editing and maintaintance.

Here's a full example:

In Netbeans ( I'm using 6.8 Beta ), go to File->New Project, Java Application, Finish
Right-click on "Source Packages"->New->Java Package, specify your package name, example "org.example"
Right-click on "org.example"->New->JDialog Form, specify a name, "ExampleDialog", "Finish"
You're done with this Netbeans project; you can now close it
Now, outside of Netbeans, copy ExampleDialog.java and ExampleDialog.form into your Eclipse project.
Inside Netbeans, File->Open File..., specify the location of ExampleDialog.java in your Eclipse project.
You can now edit the form as you would normally.

Here are a couple of caveats to keep in-mind:

When running in this mode, I've sometimes noticed a bug in Netbeans where the palette doesn't show up. If this happens, you can workaround by adding components in the "Inspector" window. (It has a right-click menu item "Add From Palette").
Eclipse doesn't prevent you from editing the "Do Not Edit" sections of the generated java, so be careful not to edit them.

The following link provides a good tutorial on using Matisse:

http://www.javapassion.com/handsonlabs/nbguibuilder/

Parallelizing Data Compression Using DataRush

yangyou — Thu, 12 Nov 2009 05:09:00 GMT

Compression has been a traditional computation task ever since the creation of zip and other common compression algorithms. The idea is to trade CPU time (normally a small slice) for potentially huge disk storage savings. This is particularly true for text data, as character streams do compress very well, i.e. the compression ratio is very high (~20% of original size). However, as data becomes ever larger and larger, the CPU time for compressing them increases. Certain algorithms (such as rar) allow for differing levels of compression, resulting in some extremely CPU intensive compress process when choosing "maximum compression". Bzip2 is a highly effective and common algorithm that typically achieve nearly 20% smaller sized outputs than gzip, however, at a cost. According to wiki, it is "considerably slower (~12 times vs Deflate on typical data)".

To alleviate this problem and to accelerate the compression process, a natural path is through parallelization. There is a surprising scarcity of known parallelized implementations in this area. As far as I know, only bzip2 and 7zip have implementations that are parallelized. In the case of 7zip, up to two CPU cores are utilized. The primary hindrance is likely in the difficulty of modifying existing algorithms while maintaining compression ratio and output compatibility with decompression methods. Most algorithms require a large window on the input data and process them one chunk at a time. Going inside the multistep compression method to parallelize can be daunting and error prone. Because the entire file is handled as a coherent whole, even if data were to be broken up, the resultant compressed stream can no longer be easily assembled together into a final piece that can be understood by decompressors.

An effective alternative is parallelizing by treating input file as a concatenation of multiple smaller files. When each of the partitions are fed into separate compressors, they operate as if on a complete file. At the output end, those compressors' output would be concatenated together, in the order of their original placement, back into a compressed file. The catch is, the decompressor must be able to understand a concatenated file and handle chunk boundary and assembling of pieces together. It turns out that for zip and bzip2, this works very effectively.

DataRush library has built-in parallelization support, because that's what DataRush is built for :). The splitting of data file into nearly equal chunks can be considered horizontal partitioning, and one simply need to round robin the pieces to the "worker threads", dishing out as many chunks as there are workers at the same time, until the entire file is processed. To achieve best results, the number of partitions would roughly be equal to the number of CPU cores on the system. To accommodate for memory usage and disk IO, a suitable chunk size is chosen, typically around 1 MB for bzip2, for example. Then, a Java IOStream based compressor (gzip from JDK, and bzip2 from Apache Commons Compression) can be working in parallel to produce the compressed output. At the output end, a roundrobin unpartitioner stitches the pieces together. By reusing existing compressors we adhere to standards and save efforts in avoiding "reinventing the wheel".

To profile the performance of the DataRush solution, a 2GB text file is compressed using 1 to 16 parallel processes on a 16 core x86 PC, with regular SATA hard disks connected to onboard controller. The execution times (including writing the 360MB compressed output to disk) are recorded. As a baseline to note, the single threaded bzip2 native binary took 9m18.839s = 559s to complete the process. Utilizing multiple cores and partitions, we were able to cut the compression time to 78 seconds. It's noteworthy that perfect and ideal parallelization is not achievable because one still need to read the input file sequentially, and put together the partitions sequentially in the order of input.

There is a performance hit from the Java Stream based implementation to the natively compiled C/C++ compressor. The former takes 652 seconds while the latter 559 seconds to compress the 2 GB data file, a 16.6% disadvantage. However, with the help of DataRush's parallelization, we were able to make use of more CPU cores and achieve near linear scale up, to a much faster run time of around 78 seconds. Run times are listed here:

Presentations Show Strength and Speed of DataRush Performance

livey — Tue, 10 Nov 2009 22:38:00 GMT

It has been a busy few weeks for Pervasive DataRush. First, we headed to Predictive Analytics World in Washington D.C. where our collaborative work with the University of Texas at Austin on predicting customer behavior for marketing and sales optimization was selected for presentation. Our presentation titled “Churn, Baby, Churn: Fast Scoring on Large Telecom Dataset” received the “Best Title” award.

Srivatsava Daruru, from UT Austin, presented churn prediction using a dataflow computational model comprised of 50,000 customers in a 1.6GB test set and 15,000 heterogenous variables. The model used horizontal and vertical partitioning to compute many divergences in parallel, which demonstrated runtime scalability both with the number of cores and size of the input dataset. “It’s impressive since it only takes 3 minutes on a commodity 16-core server, which equals a scoring runtime of 3.6 milliseconds per customer”, said Nena Marín, Ph.D., Chief Scientist for Pervasive DataRush.

Pervasive DataRush also exhibited at PAW and showed how choosing DataRush to enable the next generation of analytics saves time and money. As you can see in the graph below, the red line depicts using DataRush on commodity hardware and provides unprecedented results over competitors…up to 500x faster runtimes!

After PAW, it was a hop, skip, and a jump to IntegratioNext here in Austin, Texas. Ryan Templeton expanded further on the graph above and explained how DataRush manages high data volumes with shorter runtimes using a dataflow implementation.

Dr. Cosgrove, of The WellMax Center for Preventive Medicine, also presented “Processing Large Datasets to Improve Lives by Identifying Onset of Disease before Symptoms are Present” at IntegratioNext, which highlighted collaboration with Pervasive DataRush to develop a predictive health application that identifies diseases before symptoms are present. WellMax selected Pervasive DataRush because of our ability to analyze highly dimensional datasets quickly. Dr. Marín says "The preliminary results presented were based on a “whole patient view” to derive insights into disease precursor patterns that help us live longer and better."

Audios of all the IntegratioNext presentations should be available soon. Please send an email if you're interested in receiving audios.