Pervasive Community Site

We have moved

ijgarcia — Tue, 15 Nov 2011 15:58:00 GMT

We have moved our blogs here.

Making The Most Out Of Your Data: Big Data Opportunities

David_Inbar — Mon, 31 Oct 2011 13:46:00 GMT

Last month, Forrester Research released a report, "Expand Your Digital Horizon With Big Data,” for CIOs that focused on how they should approach big data in order to take full advantage of it for their businesses. It addressed how big data is influencing markets across industries and is prevalent in various business sectors, such as healthcare, web marketing and telecommunications. The report also discussed several factors for companies to consider when working with their data, such as redefining approaches to using data beyond traditional BI tools.

Key points included how major potential challenges of big data include not only the cost of the technology but also the shortage of data scientists. Companies are starting to seek professionals with big data skills, stimulated by new pressures to scale large volumes of unstructured data. According to a recent Forbes article by Quentin Hardy, the “data scientist” phenomenon has recently started to emerge in big data conversations with debates currently over its definition, the skill set required and the data scientist’s role in the big data trend.

Forrester analyst Brian Hopkins, who co-authored the report with Boris Evelson, Sharyn Leaver, Connie Moore, Alex Cullen, Mike Gilpin and Mackenzie Cahill, posted a brief summary of the report on his blog, emphasizing three key questions for companies to address in order to understand and create a big data plan: 1) What is new about big data? 2) What is it? and 3) How will it influence our market?

Some interesting data points, findings and recommendations from the report include:

1. Forrester surveyed 60 of their clients who are using or experimenting with big data computing. 75% of the surveyed clients responded that data volume was the main reason for looking into big data solutions. 58% of respondents in Forrester's June 2011 Global Big Data Online Survey reported interest in insight driven by an analytics approach.

2. 70% of respondents expressed interest in big data for managing current enterprise information. Therefore, many early adopters are using big data solutions to understand existing information and not new data sources.

3. Cost is the underlying theme for big data challenges. These challenges include:

· Volume, in terms of data amount exceeding how it can be stored cost-effectively

· Velocity, in terms of processing data fast enough for businesses to respond and adapt rapidly

· Variety, in terms of integration costs of adding new data feeds and interpreting variable data structures

· Variability, in terms of complex and highly variable data structures that complicate analysis

4. Big data is still in its early stage, and thus serves as a challenge to businesses. To succeed and overcome the challenges, businesses should:

· Encourage collaboration between business and IT

· Create new, agile and compliant processes and approaches to deliver big data solutions

· Adapt quickly to fast-paced, changing technology trends for rapid growth

As the big data trend continues to flourish throughout industries, companies that tend to have a handle on managing and using their data are able to develop forward-looking strategies and gain a competitive advantage over their competitors. CIOs are facing increased pressure to figure out ways to make use of the company’s data to achieve meaningful insight for valuable business decisions. As a result, this brings up an important question to address: how is the CIO’s role evolving as new big data technologies emerge and IT spending in big data increases? Is your business feeling the pressure to join the competitive big data crowd?

Pervasive Big Data encompasses Pervasive’s DataRush big data software platform for companies to consume high volume and variable data for complex analysis. We have been watching this big data trend grow for the past two years and have built our tools to help with the challenges of processing and analyzing big data.

Is Your Big Data Problem Solved Yet?

livey — Thu, 20 Oct 2011 15:21:00 GMT

We’re certainly more informed these days. And even my grandmother is hearing of the challenges of big data. But after listening to DM Radio last week, Avoiding Bottlenecks and Hurdles in Data Delivery, the big data crisis appears to be subsiding according to Philip Russom. Or is it?

Russom points out that the biggest bottleneck in big data is moving the data from processing into data flow. Old ways of processing data relied on the hardware. The faster the hardware then the faster the processing…right? Pervasive’s Big Data Director David Inbar pointed out that old software IS the bottleneck. Organizations are throwing more hardware to handle slow processes when it’s poorly written software causing the problem.

So we have organizations that have overhauled their technology to stay ahead of the curve, but they haven’t updated their software to process in parallel with their new technology. Do they still sell single-core work machines? If it’s more than a single core machine then your software needs to process in parallel for the most high performance efficiency. Parallel processing is the ability to carry out multiple operations simultaneously. And parallel processing allows organizations to expand and handle bottlenecks of data traffic. Once you’ve parallelized your working environment, is your big data problem solved? Probably not.

I’m not sure if the big data crisis is subsiding OR if big data awareness is more prevalent. Organizations are certainly starting to experience the IT side of big data, but what about educating the persons running the analytics? When Jeff Kelly wrote Data Scientists Are Rocking the Big Data World, he mentions that there are few formal training and educational programs that focus on big data and analytics. To get the right answer, you have to ask the right question. And to ask the right question, you have to understand the quality of your data.

Hopefully parents in the high tech industry are coaxing their children and their educational system to jump on the big data train today because I have a feeling that it’s going to be a long ride.

Webinar: Big Data and Hadoop with guest speaker Jim Kobielus

aschmidt — Wed, 31 Aug 2011 20:27:00 GMT

To help enterprises learn more about big data and how it fits with their traditional data warehouse and data mart, Pervasive DataRush and Karmasphere are hosting a webinar titled, "Big Data: The Role, Value and Best Practices of Hadoop," taking place September 7 at 9:00 a.m. PDT/1:00 p.m. EDT (for the Americas), and September 8 at 3:00 p.m. BST (for Europe). During the webinar, guest speaker James G. Kobielus, senior analyst with Forrester Research, Inc., will discuss the concept of big data, and the emerging software platform used for big data: Hadoop. Following the discussion, Pervasive DataRush and Karmasphere will each give a short overview of their big data offerings.

Attendees can expect to learn:

- Where Hadoop adds the most business value.
- Best practices for using Hadoop.
- How companies are combining Hadoop with standard data warehouses and data marts.
- The Forrester model of the Hadoop ecosystem.

It should be a lively discussion. If you would like to attend the webinar, please register here: http://lp.pervasive.com/BigDataWebinarWithJamesKobielus.html

BIG Thoughts on BIG Data

livey — Wed, 03 Aug 2011 20:59:00 GMT

We had the pleasure of meeting with David Linthicum last month to get his thoughts on Big Data.

Many organizations are facing the reality that they have more data coming in than they can process data. Apache Hadoop offers a big opportunity for businesses, but, according to David Linthicum, enterprises are in an experimental phase with Hadoop right now trying to learn how it works and how they can best use it.

The open-source yellow elephant has spawned a large following. Yahoo! has recently launched a Hadoop-spinoff company called Hortonworks as a prime player in the Big Data game to launch enterprise-quality products. Hortonworks presented recently at OSCON Data 2011 where they revealed their current goals on Hadoop security and framework scalability.

Software vendors are working on filling in the gaps in Hadoop to solve business problems for the enterprise. Cloudera is aiming to make the Hadoop experience pain-free with its fully integrated Hadoop distribution (CDH) and management suite; Karmasphere provides big data analytics solutions; Pervasive DataRush optimizes Hadoop jobs to increase performance on any platform.

Where is Hadoop going? We are paving the path as we speak. Stay tuned to this blog site for further details. And follow the latest industry news on Pervasive's Big Data Digest!

Capture All Your Data, All The Time

livey — Wed, 20 Jul 2011 02:44:00 GMT

We are in Washington D.C. this week exhibiting at the FOSE show and we're showing this awesome demo that we've been working on. Pervasive has taken Pervasive DataRush to new levels in order to meet specific cyber security challenges. It’s no secret that we’ve entered the big data era. There are millions of devices generating data every second: log events, security events, network traffic, firewalls, and so much more (this various is shown as shapes in the diagram below). And there’s lots of great software out there to look at these events, but only for a short time frame. One of the most daunting challenges facing organizations today is capturing, archiving, and analyzing ALL this data at any given time. There’s so much data that today’s software is failing to archive and analyze cyber security events as a WHOLE.

Until now.

Pervasive has developed a Historical Event Processing proof of concept (POC) that leverages and exposes the power of Pervasive DataRush as it captures and archives one million events per second into Hadoop’s HBase. Holy smokes, that’s amazing! This consumption rate is orders of magnitude faster than any solution on the market today.

For this POC, we used a single server box with 48 cores, 40 drives, and 258G of memory. But the processing rate will increase with increased cores or multiple nodes. We used Pervasive DataRush listeners for multiple log events and archivers to write to any database. And we actually captured 1.6 million events in 62 seconds, to be exact. Once the millions of events are captured and archived, you can use Pervasive DataRush to launch any set of queries or apply data mining algorithms to perform deep analytics on the dataset as a whole to look for a change in patterns...particularly useful in cyber security. For this POC, we used a Hive query to count server process types that generated each event and calculate percentages. Once the query was completed, Pervasive DataRush sent counts of each message to Google Charting to display the data visually. The entire process helps meet the demands of today's cyber security challenges, but we're especially impressed about the speed that we're able to capture this events and run queries.

If we captured a million events in one minute, imagine how much data is created everyday. Now organizations can capture all the data, archive it, and perform deep analytics as a whole. Stop by our booth #1528 at FOSE to see the POC in person.

Pervasive Not Only Presenting But Introducing Pervasive TurboRush for Hive at Hadoop Summit

Richard Maddox — Wed, 22 Jun 2011 14:46:00 GMT

Pervasive Software’s Chief Integration Technologist Paul Dingman will be a presenter at Hadoop Summit 2011 on June 29, 2011, in Santa Clara, CA. Paul is participating in the Application and Research track and will present from 1:15 pm to 1:45 pm PDT. Paul's presentation is "Hadoop on a Personal Supercomputer."

He will discuss the potential advantages of running Hadoop on single many-core machines with large disk arrays, illustrating cases where Hadoop running on one, or a few, fat nodes can deliver faster results and be more cost effective than Hadoop running on a greater number of lower end machines. He will also discuss opportunities to exploit intra-server parallelism to improve task communication and coordination overhead.

The Yahoo Hadoop Summit Agenda is now posted—and you can register online.

Big News to Expect at the Hadoop Summit: We’ll Introduce Pervasive TurboRush™ for Hive

As more companies adopt Apache Hadoop software for big data preparation and analytics, many are using Apache Hive for its ability to run SQL-like queries and analyze large datasets stored in the Hadoop file system.

We saw the opportunity to provide a “turbocharger” to make Hive queries run more efficiently, without developers having to modify them or learn another tool. At the Hadoop Summit the Pervasive DataRush team will introduce Pervasive TurboRush for Hive. Queries run faster on less hardware, all without code changes. Stay tuned!

You’ll also hear news about the Pervasive DataRush, Community Edition for Hadoop – a version of Pervasive DataRush for developers doing prototyping in Hadoop and looking to get better performance in their Map/Reduce cluster jobs. If you’re prototyping a Hadoop job and you’re sensitive to scaling and containing costs, this edition is for you. We’ll offer users a managed community site with forum, as well as support.

And Don’t Forget: BigDataCamp the night before the Hadoop Summit

The evening before Hadoop Summit 2011 users of Hadoop and other big data-related technologies will exchange ideas in a fast-paced, peer-driven “unconference.” BigDataCamp will be a knowledge transfer and networking opportunity for data engineers, enterprise architects, developers, analysts, data miners and business intelligence professionals. Led by CloudCamp's Dave Nielsen, attendees will share their thoughts in open discussions with pre-defined and majority-vote topics, including best practices in application development and advanced analytics. Pervasive DataRush Chief Technologist Jim Falgout will present a brief lightning talk, as will Concurrent Founder and CTO Chris Wensel, Foursquare Engineer Ben Lee and others.

Supported by Pervasive DataRush, Aster Data, Concurrent and EMC, BigDataCamp will be on June 28, 2011, from 5:30 PM-10:00 PM (PT), at the Network Meeting Center at Techmart.

Tickets are going fast so you may want to register now.

Big Data Analytics Digest: Receive the latest news, thought leadership and Pervasive Big Data Analytics activity through RSS

Richard Maddox — Fri, 17 Jun 2011 22:12:00 GMT

If you’ve explored our website lately, you may have noticed that we now offer a daily news aggregator called Big Data Analytics Digest. It’s accessible on our home page, too.

We avidly follow the torrent of news and commentary in the realm of Big Data Analytics, so we rolled up our sleeves and created the digest to give our customers, industry watchers and analysts a glimpse of what we find interesting in the ever-changing Big Data Analytics scene—including breaking Hadoop-related news.

So, if you subscribe, what will you get?

Links to the latest Big Data news from the likes of BeyeNetwork, Computerworld, Enterprise Irregulars, Forrester, GigaOM, Infoworld, IT Business Edge, O’Reilly Radar, ReadWriteWeb, SDTimes, TechCrunch, The 451 Group and others.

Thought leadership from the leading names in Big Data Analytics, including Pervasive DataRush Chief Technologist Jim Falgout and Pervasive CTO Mike Hoskins.

(FYI – if you haven’t had a chance to read Jim’s widely read article on enhancing existing applications with embedded analytics in eWeek, take a look.)

A source for upcoming Big Data industry events and presentations

Technical resources

Updates on the companies and products shaping Big Data Analytics

Pervasive DataRush news PLUS updates from our Innovation Labs

In addition, we offer a comprehensive blogroll that will keep you in touch with the leading bloggers on the Big Data Analytics market.

I’m open to all suggestions and criticisms to make the digest even better. Just drop me a line at joe.dubin@pervasive.com.

Joe Dubin,
Pervasive DataRush Product Manager

Pervasive DataRush Chief Technologist Jim Falgout to Speak at AMD Fusion Summit on June 14

Richard Maddox — Fri, 03 Jun 2011 17:20:00 GMT

Jim Discusses Leveraging Multicore Systems for Hadoop and HPC Workloads

Check out Pervasive DataRush Chief Technologist Jim Falgout at the AMD Fusion Developer Summit June 13-16 at Meydenbauer Center in Bellevue, WA. The Summit includes more than 90 technology sessions across eight Technology Topics. AMD tech leaders, industry experts, and members of academia lead the sessions.

Jim’s presentation, Session 1421 slated for Tuesday June 14th at 5:15 pm, will center on the critical importance of unlocking the parallelism of multi-core servers and clusters, particularly Hadoop-based clusters, to solve big data problems. Despite the promise of scaled-out hardware, users are encountering long processing times and complexities building MapReduce jobs.

Jim argues that the right approach to solving these challenges is to better exploit the performance potential of multicore. His presentation includes examples of Hadoop and HPC workloads running on clusters and single multi-core servers—including web analytics and bioinformatics—detailing the performance and energy efficiency gains that are possible with scaling up, as well as out.

Learn more about the technology behind his approach.

As mentioned, the Summit covers eight technology topics including:

Developer Tools: Covers development tools ranging from compilers and debuggers to performance visualization tools. Sessions cover the state of the art in compiler technology (CPU and GPU), debugging and profiling OpenCL™, and automatic data movement.

Enterprise Computing: Features sessions that discuss using multicore technology to handle large data, showcase software being developed today utilizing multicore CPUs, and show early work of applying the data parallel capabilities of GPUs to databases.

High Performance Computing: Presents a sampling of portable and standards based heterogeneous computing. Come see innovative uses of GPUs, extreme optimizations, power efficient implementations, benchmarks, libraries, and real world applications in physics, chemistry, finance and rendering.

Multimedia Processing: Sessions on image processing, audio processing, video processing, telepresence, video quality enhancement, computer vision, transcoding, content recognition, image retrieval, multimedia algorithm optimization for parallel processing and codecs.

Professional Graphics and Visual Computing: Focuses on various areas of visual computing, including mixed-mode OpenGL/DX/OpenCL™ interoperability, and advanced rendering and compute techniques.

Programming Models: Showcases the state of the art in parallel programming models and techniques for heterogeneous platforms. Topics covered include: programming models for next generation GPU architectures and techniques for building domain specific languages on heterogeneous platforms.

Security: Sessions on password recovery and audit, encryption, and steganography detection.

User Interface and Media Experiences: A focus on gesture recognition, touch recognition, face recognition, UIs for new user experiences, video management, video playback, and Web user experiences.

There’s something for everyone. The Pervasive DataRush team hopes to see you at the AMD Fusion Developer Summit!

A Detailed Summary of McKinsey Report on Big Data

Richard Maddox — Thu, 02 Jun 2011 17:14:00 GMT

We wanted to share a detailed summary of the report on Big Data by McKinsey Global Institute (MGI) that was released last month, as it contains relevant points and interesting statistics.

The report describes the state and growing role of digital data that has now entered every sector and economy, as well as the impact of the growing amount of data. MGI claims that there is strong evidence that Big Data can contribute significantly to national economies, creating substantial value for the overall world economy. Their research suggests that the public sector can increase its productivity through effective use of Big Data. For instance, the value to the US healthcare system could be $300 billion a year, and US retailers could boost their operating profit margins by 60 percent. However, MGI notes the challenges organizations face with reaching the full potential of Big Data, such as limited analytical and managerial talent to make big data advantageous and valuable for businesses.

Some relevant key findings from the research include:

1. MGI estimates the new data stored by enterprises exceeded 7 exabytes of data globally in 2010. In addition, the new data stored by customers around the world exceeded 6 exabytes in the same year.

2. Organizations are using Big Data analytics more to make decisions by analyzing datasets, including from mobile and social networks, on customers, employees and sensors embedded in products. This is leading to innovation of new business models, products and services. For example, there will be a better match between products and customers.

3. The use of Big Data will encourage new growth opportunities and competition among businesses.

4. Some sectors are positioned for greater gains from the use of Big Data. Those sectors include computer and electronic products, financial and insurance, and government. Public sectors, such as the education vertical, have experienced negative productivity growth due to high systemic barriers.

5. Concerns over the use of Big Data include data policies, privacy issues, developing new techniques and technologies, organizational change and talent, access to and integration of information from various data sources, and industry structure.

6. Under techniques for analyzing Big Data, MGI listed various methods from statistics and machine learning for data mining, including association rule learning, cluster analysis and classification. Data fusion and data integration are also significant techniques that allow analysis of data from multiple databases to extract valuable insight.

7. Visualization supports Big Data and is becoming more important to portray information in a consumable way for people to understand.

Please see below for a direct link to the full report. The report includes an overview of Big Data techniques and technologies, the potential of Big Data and key findings in five types of verticals (health care, public sector administration, retail, manufacturing and personal location data), and the implications for organization leaders and policy makers.

McKinsey Global Institute, May 2011
Big data: The next frontier for innovation, competition, and productivity

Tackling Big Data is a Big Job

livey — Tue, 10 May 2011 14:37:00 GMT

Tackling big data is not a job that is only going to be solved by programmers alone. It’s going to be solved in concert with data scientists and analysts. What tools exist for both programmers and non-programmers? We love KNIME for this and we love DataRush for KNIME even more.

KNIME, an open-source data mining and visualization tool, allows users to visually create data flows, execute selected analysis steps, and later investigate the results through interactive views on data and models. KNIME’s drag-and-drop interface gives non-programmers a chance to easily build work flows and store the analysis process for later expansion. DataRush for KNIME allows big data analytics to be processed in a faction of the time without writing scripts.

This Wednesday, May 11, Pervasive DataRush and KNIME will host a joint webinar, Scalable Data Analytics: Tools for Big Data Mining and Visualization, that will show KNIME’s intuitive user interface and demo how DataRush accelerates KNIME’s already powerful data mining capabilities. Michael Berthold, CEO of KNIME, will show interactive analytics and visualizations produced with the KNIME platform. With DataRush for KNIME, Davin Potts will show how anyone who can use a mouse can now use DataRush to maximize the full power of cores in their server without needing to be a programmer.

Making available the power of DataRush through a tool like KNIME means empowering data scientists, analysts, and programmers alike to help tackle big data analytics. If you miss the webinar, please email us for a recorded version of the live event.

Hitting the East Coast this week: GigaOM Structure Big Data 2011

livey — Mon, 21 Mar 2011 15:12:00 GMT

Pervasive Software is a proud sponsor and exhibitor of GigaOM’s Structure Big Data 2011 taking place on March 23 in New York City. As noted on their website, The Structure Big Data conference is designed to get you up to speed on how to make money using the data already locked in your organization.

In our booth at GigaOM Structure Big Data, we’ll be showing the latest release of Pervasive DataRush, a software framework for building high-performance applications for big data. Organizations can develop big data applications that run faster, cost less, and use less energy – whether on a cluster or a single server, on premises or in the cloud. Check out this video of Jim Falgout discussing the latest version of Pervasive DataRush and where it can be used. Not only are we exhibiting, but our CTO and general manager of Integration Products, Mike Hoskins, will participate on the panel titled, “The Many Faces of Map Reduce - Hadoop and Beyond,” taking place at 1:30 p.m. EST. Mike will be discussing parallelism and big data. Mike is always atop the latest industry trends and will be available to meet and discuss any questions you may have. If you’re not already registered and would like to attend, you can save $100 off registration by using this link.

GigaOM is sure to be a success and we hope to see you there!

Takeaways from the O'Reilly Strata Conference

Richard Maddox — Fri, 18 Feb 2011 21:09:00 GMT

Attending the O'Reilly Strata Conference, I received lots of food for thought about the future of Big Data, as well as further validation that Pervasive DataRush^TM is a good framework to respond to many of the information explosion challenges now or soon to be facing organizations. Here are some of my takeaways from this insightful event.

Joe Dubin
Manager, Product Marketing
Pervasive DataRush

Information is Black Gold

Metamarkets CTO Mike Driscoll told technology executives to think ‘oil' when it comes to information. Driscoll, quoting Gartner's Peter Sondergaard, stresses, "Information will be the ‘oil of the 21st century'. It will be the resource running our economy in ways not possible in the past." Driscoll describes Big Data as the ‘tar sands' of the information economy, containing valuable stores of information, but that are expensive to extract. Once extracted, the challenge is to analyze the data, using it to learn and predict.

Driscoll sees three major forces driving Big Data:

Ubiquitous sensor networks (mobile phones, as an example).
Cloud computing obviating the need to manage compute power. Drawing an analogy to an electric grid, Driscoll says that businesses don't invest capital in power generation, and the cloud enables a similar trend for compute power.
Machine learning, with Driscoll citing the progress made in the DARPA grand challenge and the Netflix prize.

Other emerging trends Driscoll has a pulse on are:

The Need for Data Scientists: Already in short supply, the demand for data scientists is growing. Companies are looking for those with interdisciplinary skills in math, statistics, bioinformatics, physics, programming (and hacking) skills, and, above all, curiosity. In fact, many speakers ended their presentations with a message to data scientists: We're hiring.
The Rise of Data Publishers (i.e., the reassertion of control by data producers): Companies recognize the value of their own data and are pulling back from third-party data processors.
The End of Privacy (or the Rethinking of Privacy): The view that visibility of personal data can be restricted is shifting to one inclined to restricting allowed usage of that data. In other words, policing usage will become more prevalent.
The Rise of Data Start-ups: A class of companies is emerging whose supply chains consist of nothing but data. Their inputs are collected through partnerships or from publicly available sources, processed, and transformed into traffic predictions, news aggregations, or real estate valuations. Data start-ups are the wildcatters of the information age, searching for opportunities across the data landscape.

Data science, Driscoll firmly believes, can solve big problems to organizations-namely, making sense of the world and scaling-up decision making. As a case in point, he cites the use of data mining to reduce health care costs by identifying the neediest patients and improving their health care.

Pervasive DataRush, Parallelism, Big Data and Hadoop are Top of Mind in Upcoming Pervasive Presentations

Richard Maddox — Fri, 28 Jan 2011 23:07:00 GMT

January, February and March will be busy months for our technology evangelists. Pervasive DataRush Director of Product Management Davin Potts, Pervasive DataRush Chief Technologist Jim Falgout and Pervasive Software Chief Technology Officer Mike Hoskins will be speaking at major conferences over the next few weeks.

At the O’Reilly Strata Conference (www.strataconf.com/strata2011) in San Francisco, running from February 1-3, Davin’’s “Supercharge Development and Performance of Hadoop Applications” presentation will take place on February 2 at 2:30 p.m. PST in the Mission City B4 room. He will discuss how to get more done faster with high-performance MapReduce and expand the universe of Hadoop possibilities with tools to speed and simplify development and deployment of analytic applications, such as Pervasive DataRush™. Pervasive Software also is exhibiting at the Strata event—you can find us at booth 501.

For those in the Austin area, on January 29, Davin also will be speaking at DataDay Austin (www.datadayaustin.eventbrite.com). He will discuss “Reducing Complexity and Increasing Efficiency in the Land of Hadoop.” We hope you can join him.

Pervasive Chief Technologist Jim Falgout will be addressing the European Data Innovation Summit (www.europeanintegrationsummit2011.com) audience in London at the Hilton London Tower Bridge on February 2. Jim’s presentation will be “Best Practices in Building Custom Applications for Big Data.” He will describe how the dataflow-based Pervasive DataRush framework delivers massive built-in parallelism to power applications that automatically scale up to consume the full capacity of commodity multicore servers.

Jim also will be speaking at the KNIME User Group Meeting (www.knime.org/about/events/ugm-workshop-2011), being held in Zurich on February 28-March 4. Stay tuned for more details.

At the GigaOM Structure Big Data conference (http://event.gigaom.com/bigdata/) on March 23 at Pier Sixty in New York City, Mike Hoskins will be part of the Hadoop panel that will be gathering at 1:30 p.m. Mike will be discussing parallelism and Big Data. Mike is always a font of industry knowledge and an astute trend predictor (and entertaining to boot).

We look forward to seeing you—no matter the time zone--in the weeks ahead!

Intel’s Concurrency Checker confirms powerful scalability of Pervasive DataRush & Pervasive DataMatcher

Richard Maddox — Mon, 17 Jan 2011 23:17:00 GMT

The Intel Concurrency Checker can be used to evaluate the performance scaling of applications on multi-core systems and to help further optimize applications. It’s a tool that is used to check application threading and threading concurrency and can also be utilized to measure performance by running the application before and after making specific code enhancements and comparing the measured results.

Computed Scaling
Computed scaling, or concurrency level, is Intel’s measure to predict the performance improvement factor when an application is run on a multi-core system, compared to a single-core system. The concurrency level is measured over a 30-second interval. Pervasive Software recently used the Concurrency Checker to conduct software performance testing of Pervasive DataRush v4.4 running the MalStone B benchmark and Pervasive DataMatcher 5.0.

Pervasive DataRush 4.4 running the MalStone B Benchmark
MalStone benchmarks, developed by the Open Cloud Consortium, provide a method for assessing data-intensive application performance for cloud-based clusters. MalStone datasets consist of information about web site visits and cyber infection status. The benchmarks calculate the rate of infection for each site (an anomaly that might signal intrusions or attempted intrusions).

Largely a ‘Read’ operation, the result shows Pervasive DataRush’s ability to tackle I/O intensive data activities with phenomenal throughput.

System Information:
Cores: 16
Processor: 164 Family 6 Model 15 Stepping 11 Intel Xeon CPU E7330 at 2.4GHz
Operating System: Windows Server 2008 R2 Standard Edition (build 7600), 64-bit
Sockets: 4
Logicals: 16

Pervasive DataMatcher 5.0
On the same 16-core system, Pervasive DataMatcher 5.0’s computed measured value was amazing.

The test serves as another proof point about CPU- and process-intensive capabilities of Pervasive DataMatcher, which allows users to fully utilize all of the capacity of their multicore systems.

Intel Case Study

Pervasive’s testing results further validate the computed scaling performance of our highly robust Pervasive DataRush-based products. Candidly (and no pun intended), our results were off the scale. The results, even more, serve as additional substantiation that the multicore reality is here and now.

As a developer or other professional seeking software capable of tackling big data challenges, we encourage you to check out Pervasive DataRush yourself. The Pervasive DataRush team also invites you to read Intel’s case study of our use of the Concurrency Checker. Look for the case study soon.

*Based on 30-second elapsed time.

Fully exploit your servers to meet analytic challenges on growing data sets

Richard Maddox — Mon, 06 Dec 2010 19:04:00 GMT

We’ll show you how on Dec. 8

Like most software organizations yours probably needs a cost-effective approach to deliver analytics or other data-intensive solutions amid increasing data volumes and growing processing complexity—one that allows you to enhance your ability to gain maximum performance from existing servers for data-intensive jobs.

Scaling performance, of course, will be a key to your success. How can you get it?

Multicore processors from Intel, AMD and IBM redefine the compute density and capabilities of commodity servers. Now, rather than relying on endless scale-outs, you can scale up within every server in your cluster and achieve radical throughput gains when you prep, process and analyze Big Data.

On December 8 at 12:00 p.m. CST/1:00 p.m. EST/11:00 a.m. PST, join us for an instructive webinar presented by Pervasive Software CTO Mike Hoskins and Pervasive DataRush Chief Technologist Jim Falgout to learn how to quickly build high-throughput analytics software that will give you eye-popping performance on multicore servers.

Mike and Jim will detail how to:

» Deliver 20+ times the throughput at a fraction of the cost of deploying or expanding a cluster
» Quickly build your analytic models using a visual open source tool
» Avoid the multi-pass requirements of SQL and deliver high-throughput custom analytics

Don’t miss it.

You have my permission

bjacaruso — Thu, 21 Oct 2010 15:37:00 GMT

I met a software vendor recently who was describing his product and he said, "We are a real cloud app, not like Google mail or something like that." What? I guess there is an argument in the software world as to what qualifies as a 'real' cloud app and what is not a 'real' cloud app.

That sounds funny to me. It seems like if your application leverages cloud computing, then it is a cloud application. If your application resides on your desktop, then your application is a not a cloud application.

My acquaintance built a sophisticated business process management full SaaS application that ran on Microsoft's Azure platform. I guess he felt insulted when people tried to compare him to, of all things, Gmail!!! Go figure.

Gmail is quite impressive to me and certainly seems like a cloud app. Because of this conversation I have torn down the wall in my head. No longer do I try to classify a cloud app by its complexity, size or even where its code runs.

I now say that if an application leverages cloud computing elements, then you have my permission to call it a cloud application.

Cloud Computing Solves Cost, Time and Expertise Puzzle

bjacaruso — Fri, 15 Oct 2010 06:31:00 GMT

I was speaking with a colleague yesterday about our cloud offerings. We were discussing people who grew up in the on premise world and wanted to know what the big deal was. I realized I could have gone on for hours about the advantages of cloud, but I wanted to keep it short. I came up with the core business area cloud improves on is IT infrastructure in the form of cost, time and expertise. Let me explain each of those.

Cost advantages in the cloud are twofold. First because of economies of scale it is simply less expensive in terms of server to server comparisons. No matter what the sales guy at Dell or HP tells you, he cannot make it cheaper for you to own a Windows server for a month (once you roll in the underlying costs of maintenance). The second dimension of cost advantage is that my resources are now disposable. When I am done with it, I can get rid of it. I don't even need someone to come pick it up and cart it off.

Time advantages relate more to the procurement cycle. If I need a new server, load balancer, or more storage in the cloud, I make an API call. If I need more of anything in my own data center I have to make a telephone call to a sales guy. I wonder which is faster?

Finally: expertise. Granted that in any technology based infrastructure I will need experts, will I still need the guy whose sole job is to provision, update and build servers? Probably not. Cloud IaaS vendors are now even supplying a collection of preconfigured server / software packages ready to go as off the shelf images. Think about getting a hardware based server, installing both an application server and database ready to go in your shop. How many different people does it take? On the cloud I can do it by the simple click of a button!

Cloud, when compared to an onsite IT data center really does have significant advantages. When making the decision to cloud or not to cloud, be sure to take in the elements of cost, time and expertise.

Update Available: DataRush 4.4.1

azeemj — Wed, 06 Oct 2010 13:00:00 GMT

We've been working hard here at Pervasive and DataRush 4.4.1 is now available for download. Don't let the small version number increment fool you, there are many worthwhile changes to DataRush. Including new documentation that shows how easy it is to use the DataRush analytics packages. The new and updated documentation are easier to read and are more complete than before. DataRush now supports KNIME 2.2.2, and is a recommended upgrade for all KNIME users.

If you're new to DataRush, request a free trial and see what DataRush can do for you.

Did anyone notice that a 48-machine cluster is now available on a single server?

Richard Maddox — Fri, 01 Oct 2010 15:14:00 GMT

Did you notice the sea change taking place in big data computing?

Multicore processors are redefining the compute density of a single server. When fully leveraged with the right software, multicore processors can provide aggressive parallelism to tackle big data challenges, outperforming much more expensive cluster computing.

Today, 4 AMD Opteron 6000 12-core processors in a single server can be more effective, green and cost friendly than relying on a cluster. A single-box solution removes cross-machine communication delays. With a cluster, you incur additional reliability overhead to accommodate the very real case that a single node goes down; but within a single multicore box, the probability of any single core going down is negligible. In a cluster, you incur overhead distributing data across a network; within a single multicore server, no such overhead is required.

It all rests on the software you choose to build and deploy. Shouldn’t you make software design decisions based on where the industry is going – dense computing – rather than where it's been – distributed computing?

On October 7, this coming Thursday, Pervasive CTO Mike Hoskins and Pervasive DataRush Chief Technologist Jim Falgout will discuss how a single box utilizing Pervasive DataRush can deliver incredible performance while saving you the expense, overhead and maintenance of a cluster. Register now.

Pervasive PSQL v11 MC Release Candidate - But Wait, There's More

answerman — Wed, 11 Aug 2010 15:34:00 GMT

Dateline: August 11, Austin

Pervasive PSQL v11 MC Release Candidate is here. If you haven’t already downloaded the beta (and even if you have), you should definitely download PSQL v11 MC RC and give it a test. What started two years ago as a fairly quiet, under the radar type product plan has turned into a full blown big list of features release.

Part 1 - Multi-core

Raise your hand if you’ve bought a server in the last year. If you have, it’s got a multi-core processor. Minimum would be a dual core, more likely a quad, and lots of newer boxes are coming with 8-cores. Now, raise your hand if you’ve designed your application with a lot of parallel processes to run on those multiple cores. Most companies have upgraded hardware, most haven’t written their applications to take advantage of it. Many applications are going to run more slowly as a result. (Managing multiple cores, cache and synchronizing everything creates a LOT of overhead.) For more on why this happens, check out the whitepaper from CITO Research - The Multi-core Dilemma.

The good news - Pervasive PSQL v11 has been redesigned to do a lot of things in parallel and is now optimized for multi-core hardware. For the details, check out the What’s New in Pervasive PSQL v11. Better yet download a trial. Or, you could start over with your current code, find which parts can be run in parallel, solve all of the issues related to synchronizing memory access, debugging, testing, etc. Probably only take a couple of years, if everything goes perfectly. Trust me, the simplest thing to do is go with PSQL v11.

Part 2 - IPv6

The end is near. Really. The current standard for Internet addresses, IPv4, is due to run out of addresses in less than a year. Check out http://penrose.uk6x.com/ for a countdown clock. What’s next - IPv6 - which will be using 128-bit addresses (instead of 32-bit with IPv4) and should last for a ridiculously long time. IPv4 had 4 billion addresses (sounds like a lot - but with 6 billion people on the planet and lots of devices, blogs, sites, etc. we were eventually going to run out).

IPv6 has 340,282,366,920,938,000,000,000,000,000,000,000,000 - or roughly, 340 undecillion (look it up, I did) addresses. That’s about 57 trillion billion addresses for each and every one of us. We won’t run out any time soon. The issue for you is that governments (Japan and US) and other large institutions are requiring IPv6 compatibility. More good news - PSQL v11 already supports IPv6. So when the end arrives, you’ll be ready.

Parts 3 through N - PDAC for RAD Studio 2009 and 2010, ADO.NET 3.5, 64-bit ODBC driver, etc.

As usual, Pervasive Engineering has done a fabulous job of adding all kinds of additional features and updates. I start sounding like Ron Popiel (think Ronco, pocket fisherman, veg-o-matic, inventor, and the greatest TV pitchman ever) - but wait, there’s more. Here’s what else you get with PSQL v11:
-   64-bit ODBC driver
-   PDAC for RAD Studio 2009 and 2010
-   ADO.NET 3.2 and 3.5 with Entity Framework Support
-   Telephone activation
-   New releases of AuditMaster and DataExchange - and soon Backup Agent
-   Backward compatible, simple upgrade

Pervasive PSQL v11 helps you skip over some really huge technical hurdles (multi-core and IPv6) and adds a boatload of cool extra features. To get all of the details, go to the trials section of the Pervasive website and/or check out the Pervasive PSQL v11 FAQ.

It’s awesome.

Can You Define (Really) Big Data?

livey — Fri, 06 Aug 2010 21:04:00 GMT

Jim Ericson’s blog on (Really) Big Data perked our interests. In the industry of big data, we’re always curious to know what IS big data? It used to be terabytes, now it’s petabytes, and it’s exponentially growing. We agree with Jim, it’s not about how much data you have but how much valuable information you’re getting from it. As cited by Davin Potts, our Director of Product Management, “Sipping from the firehose isn't impressive. Big data is when I can scale to process the volume, keep up with the firehose, and afford to crunch through all of that data that I can get my hands on. Anything less is just a big number.” Read more.

Pervasive DataCloud2 creates success for Amazon Web Services, Microsoft Azure and Force.com!

bjacaruso — Mon, 12 Jul 2010 14:04:00 GMT

Reuven Cohen ( http://bjac.it/brPe6V ) described this morning in a couple of paragraphs ( http://bjac.it/aSDu5x ) what I have been trying to explain for the last 18 months. The Pervasive DataCloud2 is not a competitor to other clouds or Platforms as a Service, but a complementary service.

I would go a step further and say that the boutique cloud services are critical for the continued success for the likes of Amazon, Microsoft and force.com. I am sure that the guys in Seattle (both groups) and the teams in the valley ( http://bjac.it/aJqxjd ) would like to tell you that they can be everything to everyone. But they know as well as we do that is not part of their strategy. Sure, they will try to offer as much as they can in terms of variety of service, but in the end compromises will be made for the greater good of all.

Pervasive DataCloud2 and services like it add pieces ( http://bit.ly/j4n2U ) that these vendors cannot and probably should not offer as part of their platform. Can a guy like Mark Benioff really justify spending time helping a company integrate their Netsuite with an on premise SAP implementation? Hosting the service infrastructure makes perfect sense for Amazon, but should they develop the integration expertise? Should Steve Ballmer and the crew in Redmond really be in the business of analyzing an organization’s healthcare data?

I say no to all of the above, and even though they will not say it publicly, I am guessing the above list of cloud vendors know that it is a truth and fact that niche cloud services guys are a key component of their success. You are welcome guys!

Pervasive DataRush: Cost-effective security for companies in a challenging economic climate

Richard Maddox — Wed, 07 Jul 2010 22:00:00 GMT

Security spending in a downturn is under tight scrutiny. PricewaterhouseCoopers found this to be the case when it surveyed 7,200 executives in over 130 countries for its 2010 “Trial by Fire” report. One of the report’s primary findings states:

Not surprisingly, security spending is under pressure. Most executives are eyeing strategies to cancel, defer or downsize security-related initiatives.
Source: PricewaterhouseCoopers 2010 “Trial by Fire” report.

PricewatershouseCoopers notes that 70% of survey respondents think it is important to consider canceling, deferring or downsizing security-related initiatives if they require capital expenditures while 71% respond similarly for initiatives requiring working expenditures.At the same time, survey respondents overwhelmingly said they considered security strategies to be important, including increasing the focus on data protection, prioritizing security investments on risk, reducing or mitigating major risks and accelerating the adoption of security-related automation technologies to increase efficiencies and reduce cost. With many analysts predicting a slow recovery, the pressure to cut or curb security spending could increase. Meanwhile, security intrusions continue, likely at a more sophisticated scale (consider “Ghostnet” for example).

So, are there any cost-effective solutions available to meet growing security? Yes.The massively parallel-processing horsepower of the Pervasive DataRush™ data processing engine combined with our matching capabilities forms a powerful solution for translating massive amounts of raw data into actionable intelligence. Combined with Pervasive DataMatcher™, a cost-effective robust, innovative solution is available to financial, insurance, healthcare, law enforcement and homeland security organizations that want to leverage powerful, next-generation analytics for detecting fraud and corruption, complying with anti-money laundering controls, and security and compliance monitoring.

The Proof is in the Pudding: MalStone B-10 Benchmark

This year Pervasive DataRush conducted internal testing using the MalStone B. The benchmark examines large volumes of logfiles to look for anomalies that might signal intrusions or attempted intrusions.

The data file for MalStoneB is generated by a Python script and the MalStone records have the format:

Event ID | Timestamp | Site ID | Compromise Flag | Entity ID

These describe a visit by an entity to a site at a particular time. After the visit, the entity sometimes becomes compromised, which is indicated by setting the compromise flag to 1. Each record is 100 bytes. MalStoneB computes a ratio for each week d, and computes for each site w, and for all entities that visited the site at week d or earlier, the percent of visits for which the entity became compromised at any time between the visit and the end of the week d.The Malstone benchmark can use a variable sized dataset, In the experiment a 10 billion row dataset totaling 1 terabyte was used.

Summary of Details:

32 core, 4 socket, 2.0 Ghz Intel Xeon X7550
1890 seconds (32.5 minutes)
5.29 million rows/sec, approx 509 Mbytes/sec

The Result: In just 31.5 minutes, 10 billion records were searched for anomalies using Pervasive DataRush. This result is 26x faster than its competition.

Pervasive DataRush can enable applications that scale seamlessly on a single multicore server (rather than a cluster) to prepare or analyze even massive datasets – at unprecedented speeds. Cost-effective approaches to daunting security challenges – that’s what IT executives seek in the midst of a lingering downturn, and that’s what we want to give them.

Distributed, Scalable Clustering for Detecting Halos in Terascale Astronomical Datasets.

n5712036 — Fri, 02 Jul 2010 14:47:00 GMT

The process of stellar discovery has long made its home at High Performance Computing (HPC) systems.
HPC systems have evolved into clusters of "fat" multicore nodes. Applications must take advantage
of parallelism across nodes and at the node level to maximize scalability and performance/watt.
The complexity of multicore programming underscores the need for powerful and efficient runtime
systems that manage resources such as cores, threads, memory, and communication sub-systems on behalf
of the application. Dataflow is the computational model in Pervasive Datarush to construct
efficient data-parallel pipelines via threads while abstracting the complexity of multicore programming.

Simulation of the dynamical evolution of the entire observable universe via N-Body interactions
begins with the presumed first principles of the universe: cosmic background radiation; an expanding
volume of cooling helium and hydrogen; dark matter separating from gas and coalescing into massive
stars. The classical N-body problem simulates the evolution of a system of N bodies, where the force
exerted on each body arises due to its interaction with all the other bodies in the system. N-body
algorithms have numerous applications in areas such as astrophysics, molecular dynamics and plasma
physics. The Cube3PM method for carrying out large N-Body simulations to study formation and evolution
of the large scale structure in the universe combines direct particle-particle forces at small scales
with particle-mesh ones at larger scales (Particle-Particle-Particle-Mesh Method). Such an approach
produces datasets with 4000^3-5488^3 (64-165 billion) particles. Several such simulations were
completed on Ranger Cluster (Texas Advanced Computing Center) on 4,000-22,976 cores.

From the astrophysicist’s perspective, the problem of identifying regions of interest in this terascale
data and being able to visualize these regions is made intractable by the overwhelming volumes of data.
Current methods to detect individual halos fall into two basic categories, namely friends-of-friends (FOF)
and spherical over density (SO) methods. The FOF method is particle-based. Dense regions are identified
by locating particles that are closer to each other than a pre-defined distance, which is a parameter of
the model and is usually referred to as 'linking length' . Particles that are within that distance from
each other are called 'friends', and the halos produced consist of all particles which are connected by
a chain of friends. The SO class of methods, on the other hand, start by identifying the local density
peaks (or gravitational potential minima) as the halo centers and then expand spherical shells around
those centers until a pre-defined density threshold (a free parameter of the model picked based on
dynamical considerations) is crossed. Within these types of methods there are multiple variations,
regarding e.g. how the halo centers are located, how the gravitationally-unbound particles are treated,
etc. Each of the two basic approaches, FOF and SO, has its advantages and drawbacks and can fail in
certain situations (Tinker et al.).

"Automated methods for halo identification and visualization are critical to advancing the physical
understanding of what is happening through better analysis", said Astronomy Centre at the
University of Sussex, UK.

Our dataflow methods supply an alternative to the current approaches which on one hand is
density-based like the SO, but does not make assumptions about the halo shapes as the SO does.

This dataflow implementation distributes itself across multiple nodes on the Longhorn cluster.
Likewise, this parallelized dataflow AutoHDS facilitates the use of large number of cores on a
single cheap machine instead of expensive super computers. Experiments revealed that when data
points were uniformly distributed across partitions, dataflow AutoHDS achieved linear speed up
with the increase in the number of machines used. Dataflow AutoHDS also yields better performance
with increasing data volumes. In comparisons against Hadoop AutoHDS, dataflow was consistently
faster on fewer resources.

This work has been submitted for publication. Coming here soon....