Small things often come in big packages, and this is something that the enterprise world is starting to realise as it continues to adjust to the Big Data paradigm. Data analysis must be conducted before making strategic business decisions these days, and this analysis is likely to come from one of the many tools provided by the Big Data industry, a sector expected to generate sales more than $187 billion next year.
It is safe to assume that Big Data is here to stay; however, the enterprise focus is likely to shift towards reducing the size of the massive data sets being collected and processed for the purpose of perfecting the analysis and making even better decisions.
The Problem With Big Data
In the world of Big Data, one size does not fit all. Over the last few years, the enthusiasm of being able to collect infinite amounts of information has overshadowed analytical implementation. The problem with Big Data is that it has grown out of proportion for many businesses. There is no question that computer science is progressing in the right direction of Big Data; however, only a small portion of the enterprise world stands to benefit from the technical aspects of data mining.
In 2018, business owners should not assume that Big Data would drive them to success; this is an assumption improperly made by retail giants Tesco and Walmart, two data-driven companies that invested heavily in Big Data only to realise that it was not the silver bullet they envisioned.
In the case of Tesco, Big Data analytics had predicted a surefire hit with its Fresh & Easy stores in the United States, but the analysis failed to include the potential of an economic downturn for which Tesco did not have an exit strategy. In the case of Walmart, the company has invested millions in data science operations only to realise that its operating margins have only improved very slightly and certainly not enough to keep dozens of Sam’s Club stores open beyond 2018.
Why We Need Small Data
As a technological paradigm, Big Data is not going to stop; however, most of the enterprise world will only need to look at Small Data to formulate strategies. Small Data comes from Big Data. It consists of information that is presented in manageable ways that decision-makers can understand and implement.
We need Small Data because recent history shows that we have been overdoing it with Big Data. Search engine giant Google was faced with the reality of a Big Data project that did not work out as intended. In 2012, the Google Flu Trends analytical tool, which aimed to estimate influenza outbreaks around the world based on search queries, prompted clinics to stock up on medications when the GFT tool suggested the incidence of infection in certain communities. By 2014, data compiled by the U.S. Centers for Disease Control showed that Google overstated the magnitude of outbreaks by 50 percent in some cases.
An even better example of Big Data analysis gone wrong was the November 2016 election that predicted a landslide victory for former Secretary of State Hillary Clinton. One of the problems, in this case, was that forecasters took to Big Data for correlation when they should have been looking at Small Data for causation. We should not develop unhealthy Big Data obsessions at the risk of becoming overly optimistic with numbers that seem to prove everything.
But What About Big Data?
As previously mentioned, there is nothing wrong with Big Data. In fact, the huge advances in data science and its embrace by the enterprise sector is probably the most significant business development of the 21st century. Besides, we can only obtain Small Data from Big Data.
The relation between Big Data and Small Data is like ying and yang. As it applies to e-commerce, Big Data is useful when it is used to determine transaction patterns; meaning that online retail giants such as Amazon will continue to rely on the analysis of clicks, bounce rates, payment behaviours at the shopping cart stage, and website heat maps. Small Data will be better suited for Amazon as the company analyses its Whole Foods acquisition and the Amazon Go convenience stores.
With Small Data, store owners can glimpse into the communities where shoppers are coming from. The idea is to extract smaller data sets from the massive amounts of information processed by Big Data scientists, and the analysis should include experiential experiences derived from the actual act of shopping. This is a matter of gaining insight that cannot be obtained by simply looking at numbers.
When looking at Small Datasets, decision-makers should consult with business owners who are familiar with customers and their communities. To a great extent, the burgeoning trend of Small Data is an exercise in going back to basics. Regarding value, the insights extracted from Small Data must be accessible, understandable, and actionable.
Big Data is not human enough for most business organisations to assimilate and apply. If the goal of a decision maker is to collaborate with key personnel and stakeholders to develop strategies, Small Data is the way to go. In the end, the quantity of data gathered for analysis does not matter as much as what companies can do with it, and this could mean looking at relevant Microsoft Excel datasets extracted from large clusters.
