We’re certainly more informed these days. And even my grandmother is hearing of the challenges of big data. But after listening to DM Radio last week, Avoiding Bottlenecks and Hurdles in Data Delivery, the big data crisis appears to be subsiding according to Philip Russom. Or is it?
Russom points out that the biggest bottleneck in big data is moving the data from processing into data flow. Old ways of processing data relied on the hardware. The faster the hardware then the faster the processing…right? Pervasive’s Big Data Director David Inbar pointed out that old software IS the bottleneck. Organizations are throwing more hardware to handle slow processes when it’s poorly written software causing the problem.
So we have organizations that have overhauled their technology to stay ahead of the curve, but they haven’t updated their software to process in parallel with their new technology. Do they still sell single-core work machines? If it’s more than a single core machine then your software needs to process in parallel for the most high performance efficiency. Parallel processing is the ability to carry out multiple operations simultaneously. And parallel processing allows organizations to expand and handle bottlenecks of data traffic. Once you’ve parallelized your working environment, is your big data problem solved? Probably not.
I’m not sure if the big data crisis is subsiding OR if big data awareness is more prevalent. Organizations are certainly starting to experience the IT side of big data, but what about educating the persons running the analytics? When Jeff Kelly wrote Data Scientists Are Rocking the Big Data World, he mentions that there are few formal training and educational programs that focus on big data and analytics. To get the right answer, you have to ask the right question. And to ask the right question, you have to understand the quality of your data.
Hopefully parents in the high tech industry are coaxing their children and their educational system to jump on the big data train today because I have a feeling that it’s going to be a long ride.