Pervasive DataRush

Credibility is our first constraint

2008-08-12T14:42:51Z

There are always a near infinite number of challenges with launching a new product; even more so if the product is conceptually different than the others in the space. Following the Theory of Constraints, you have to focus on the current constraining issue until it is resolved and you can move on to the next one. With most new software products the constraints are tactical: code is late, functionality is limited, competition is too strong, execution is too difficult or expensive, and so on. None...(read more)

DataRush release candidate is ready for download

2008-08-08T19:56:19Z

The first release candidate of DataRush is now available to download . The main new feature of this release is the support of dataflow graph composition in Java! In the previous releases of DataRush, dataflow graph composition was accomplished through a combination of an XML scripting language and composition helpers known as customizers. This led to a quagmire of XML and Java code that was not always easy to follow. But no more! Please feel free to download the release and experiment. There are...(read more)

DataRush white paper

2008-08-07T16:18:42Z

The following is a new white paper written by our Chief Technologist, Jim Falgout. I have copied the text here, but the clean .pdf is also attached and is more complete: Pervasive DataRush™: An Architecture Paper Describing the Highly Parallel Dataflow Framework A Pervasive Software White Paper July 2008 Audience This architecture paper is written with the systems architect or Java developer in mind. It is intended to provide a high-level overview of the highly parallel data-processing framework...(read more)

Dataflow implementation in Java

2008-07-23T15:24:42Z

DataRush is sufficiently sophisticated (or at least different) that understanding it takes several passes. I am writing a series of posts aimed toward exploring what DataRush is and is not. This should give the passing programmer a better feel for what DataRush might do for them and how it fits into the broader scheme of concurrent programming techniques . So, back to the question at hand; is DataRush dataflow ? The answer is obvious: that depends upon what "dataflow" is. According to ...(read more)

Diminishing returns from virtualization will affect larger core count server sales

2008-06-23T16:33:05Z

I just got back from an HP show where I had some interesting conversations regarding the crisis that hasn't seemed to have happened yet. The key to the Sherlock Holmes mystery, "The Hound of the Baskervilles" was this: Why the dog didn't bark? Why hasn't the so-called multicore crisis been seen as a crisis? Why isn't there any angst over lack of parallelized applications and engines? Why is IT still buying multicore servers when they aren't doing development that can...(read more)

the zettaflop, the yottaflop and the xeraflop

2008-06-09T16:59:05Z

At a recent industry conference, it was shown that the biggest growth area in computing is HPC, or High Performance Computing. This was surprising, even to those in this field, as it has historically been a fairly small, insular, academic area populated by geeky professors and hard-working grad students, all using Fortran. But a new phrase has become common, HPCC, or High Performance Commercial Computing. This is an attempt to make a distinction between the academic world and the business world,...(read more)

Explaining our secret sauce - Part Two

2008-06-06T21:04:45Z

The first time I blogged on this, Explaining our secret sauce , I referenced the Wikipedia entry for "dataflow programming". I have subsequently learned that it would be more accurate to point to Kahn Process Networks as the specific type of dataflow that was the genesis for our Pervasive DataRush library and engine. "KPN is a common model for describing signal processing systems where infinite streams of data are incrementally transformed by processes executing in sequence or parallel...(read more)

Multi-threaded development joins Gates as yesterday's man

2008-06-06T16:26:51Z

"...the validity of multi threading is under attack. Veteran programmer Knuth said in a recent interview that multi threading may not be up to the task and could fail. As such, he is "unhappy" with the current trend towards multi-core architectures." “To me, it looks more or less like the hardware designers have run out of ideas, and that they’re trying to pass the blame for the future demise of Moore’s Law to the software writers by giving us machines that work faster only on...(read more)

Analysis -- the "Plastics" of today

2008-06-05T20:36:33Z

Before our current crop of graduates were born, there was the movie "The Graduate". One famous scene has our hero walking around at a party in his honor. He is given some sage career advice: "One word. Plastics". The newest member of the DataRush team, who is still a student, found the following quote and thought it was interesting and very DataRush related. Hal Varian, Google’s chief economist (and intellectual superhero), says on a Freakonomics blog post: “If you are looking...(read more)

The X=X+1 Issue

2008-06-04T21:03:15Z

A very clear explanation of why writing parallel apps is a challenge with procedural languages, versus declarative approaches. Although Java is procedural, our Java implementation of dataflow is avoids this pitfall. "The ability to assign a memory location with variable data is the cornerstone of computer programming. It turns out the ability to re-assign that same memory location is perhaps one of the biggest consternations in the parallel programming world. At the time, multiple assignment...(read more)

Multicore is most disruptive

2008-06-02T22:00:35Z

Gartner has published a list of the top 10 most disruptive technologies of the next 5 years, and multicore is number one. You can see the presentation here: Top-10 Disruptive Technologies for 2008 to 2012 Speaker: Carl Claunch & David Cearley View presentation We would probably include cheap storage as the second item on the list - Fry's has 1-Terabyte drives for $169 this week - but the combination of cheap storage and multicore power is surely a disruptive pairing. Why is more power disruptive...(read more)

Explaining our secret sauce

2008-05-30T17:32:19Z

We wish we had some competition. If there was some other vendor implementing dataflow, then we would not have the sole responsibility of teaching people about this approach. While there are certainly a large number of people who are familiar with the concept, most of the people we are meeting in presenting DataRush are not. When we explain and illustrate the phenomenal results (as in 3 hours to 22 seconds), often the first question is "How can this be?" My response has been to point them...(read more)

The tubas are blaring and the drums are pounding.

2008-05-30T17:12:27Z

I just returned from our first presentation of DataRush to potential partners, SIs, and developers. While we have exhibited at a bunch of shows like JavaOne, Innotech, and the HP Global Showcase, and while we have briefed a bunch of smart press people and analysts, this was our first time with this type of audience. Overall, we were very pleased at the interest and response to our story. The IT environment is looking for practical solutions to utilizing the multicore power that is flooding into the...(read more)

Low-Latency Technology Outpacing Programmers’ Capabilities

2008-05-30T16:45:02Z

"Again and again, executives said that finding enough programmers who are able to write "parallel" code -- programs that efficiently divide workloads across distributed processors -- is almost impossible. As Wall Street firms rely on multicore processing and even distributed computing to handle the ever-growing number of trade-related messages that are sensitive to any increase in data latency, the divergence between the capabilities of the technology and the capabilities of the programmers...(read more)

The Lawnmower Law

2008-05-21T17:10:52Z

This article is a simple illustrative introduction to Amdahl's Law. "Indeed, just as with parallel processors, there is a point of diminishing return. Adding the first 10 riding mowers reduced the time by 36 minutes. Adding another 30 only saved me 4 minutes. Adding 100 mowers makes little sense since I’ll never get below 20 minutes. (Although I would love to see such a lawn mowing demolition derby — in my neighbors yard of course.)"...(read more)