When survival is on the line, size really does matter.
As the traditionalist data analytics professionals dig their heels in and refuse to give in to the Big Data deluge, it is fast becoming clear that the volume of evidence for the new movement is too substantial to deny. There is far more data in NoSQL stacks than there was in SQL, Hadoop has long since superseded Oracle in terms of volume and data oceans are now taking the place of data lakes.
The fact that the industry still has teething troubles is not relevant it is Big, and it is here to stay.
If we consider the pace of technology over the past decade, we have seen vast advancements over such a short period in every sphere. When it comes to the Big Data industry, however, the volume of data available may well have outpaced the ability by which we can analyse it.
If you have a gigantic orange the size of the moon, you should be able to squeeze out a glass of orange juice for everyone on earth. Due to all sorts of constraints, when Big Data only squeezes out enough for your weekend Church fete, the naysayers feel justified in their criticism. If there is so much data, why cant we get more out of it?
The problem is that Data Science, in its current form, is an incredibly young industry, and we may not yet have the human expertise or technological ability to squeeze every last drop of orange juice from the moon. Having said this, this doesnt mean that we never will. It is inevitable that technology will catch up with the volumes of data available. When machine learning and A.I. are integrated into the equation, the orange juice will flow like never before.
Just because something isnt quite happening now, doesnt mean that it has no future. Also, just because your company is not seeing value, doesnt mean that others arent.
There are pockets of Big Data excellence all around the world Germany is a prime example. However, why would privately run companies want to share their bespoke techniques? It is their competitive advantage, and they have nothing to gain by letting their rivals in on the secret. It is always this way with new industries and maybe that is why there is a public perception that Big Data has teething troubles. No one really understands it (still), and no one really sees any public proclamations about its benefits.
The one thing that will save Big Data is its size.
Like the moon in the night sky, it is not going anywhere, but unlike the moon, it is certain to grow exponentially over the next decade. The analytical possibilities will catch up at some point, and then it will be orange juice all around.
