Corporate Big Data Fills Digital Trash Cans

There is one purpose of collecting data that is obvious but worth a reminder: keeping important data in-house. The more data you own, the more likely it is you build the value that comes with it. Of course, the probability function is not linearly proportional so one cannot, generally (note the exception of Santa Claus), double the value of collected data by doubling its volume.

We Expect Big Data Value Driven by Advanced Analysis

Thinking about investment in big data (whatever this term precisely means) we tend to expect that there is relatively more value in it, compared to standard data. What is the reason for that? One is, of course, complexity. We know that the value of data patterns and correlations is much more important than the data itself. Impactful correlations of data delivered from various sources are the most common examples of big data-driven added value. In an ideal world the probability function I mentioned above would therefore be exponential. However, in order to materialize this potentially surplus value, the implementation of advanced, specific analysis is necessary.

This is about revenues of big data. When we double our collection we can analyze four times more correlations, each with possible business value. And take note: even though the cost of analysis quadruples the cost of collecting just doubles. So going big data influences not only revenues, but also profits. The effect is better than linear.

Big Data Profitability Suffers Novelty Fate

Considering previous remarks why are we still cautious or event reluctant to take the value of big data for granted? It partially seems to be the same old story of pioneering concept and technology. The new ones are, at the early implementation phase, more expensive than standard, old ones. They need expensive resources, including innovative tools and people of uncommon skills. This makes the cost of innovation relatively high. Saying this I mean in relation to the good old ways of making things happen. Those old ways are standardized and widely accepted and their costs are reduced by experience and scale.

Going back to the simple example when you collect big data (i.e. new, huge sets of unstructured date derived from non-standard sources) and double your volume your cost is more than doubled. The same with analytics. When the volume of big data is doubled (and quadrupled the correlation matrix) the cost of big data analytics can rocket to unbearable heights. One simple example of operational costs is on-line access to big data warehouses. The basic requirement for effective analysis is handy and time-effective access to the entire volume of collected data volume. It is no problem when the standard data base is concerned, but the more big data is big the more complex and expensive it is to meet such requirements. Needless to say, handy data access is just the first of multiple cost-creating steps in a long chain of analytical process.

Conservative Approach – Collect Data, Delay Analysis

As a consequence of the continuously decreasing cost of data storage and unpredictably high cost of unconventional analytics, the common scheme of big data projects repeats itself. The successful implementation of big data collection, and introduction of new sources and kinds of data, is followed by ineffective use. It means the individual results could be overwhelming and perfect; just enough to be shown at professional congresses as big data business cases. At the same time the total value taken of collected data is not satisfactory. The big data projects end with partial success, but they do not impact the core business of the company, although the potential value of already stored data would make it possible.

In practice there are plenty of analytical activities performed on big data every day. But they are left unattended by senior management after the project implementation phase is over. Those activities deliver some value but, again, there is no big deal comparing the value delivered by core analytics of standard business data.

Does this picture of a continuously growing trash can of potentially valued but never used data look familiar? Sometimes reality looks even less optimistic: there are plenty of trash cans of data (each bought by separate big data initiatives) and companies cover the increasing cost of maintaining the trash. They hope it will be useful one day but this day doesnt come. This is one of the side-effects of the customer centric business approach, commonly claimed as the indispensable way of making business with modern, digital, demanding customers. It is hard to disagree that being a customer centric organization is a key survival factor today. But it also leads to the overestimation of every single customer’s data the consequences of which are increasing costs of data maintenance. Maintenance which usually brings hardly any value.

Big Data Trash Can Turn It Into a Recycle Bin

There are at least two ways to solve the dilemma of over-flowing data trash cans. The first is to limit their number and consequently erase unused data on regular basis. It is a cost-oriented strategy exploiting process optimization and careful prioritizing. The latter allows a slowing down in the pace of trash creation and is an advanced version of cost cutting. The strategy is a reasonable consequence of the lower costs of acquiring new big data than exploiting the older, which is already collected and stored.

The second focuses on putting the already collected trash to better use. I can call it a recycling strategy. In contract to the previous method this is a revenue oriented attitude. In this strategy, the continuous process of analyzing data makes it more valuable. The data is collected once and used in various models and projects. This approach, although potentially more profitable, creates more operational costs. The reason being that it needs to apply costly analytic processes several times to the already collected and stored data. But I think it is very tempting to try and recycle your data trash into valuable information before you decide to give up and forget about it forever.

I would like to thank you my friends on linkedin who helped me to create this post. However, i am the only one responsible for every linguistic imperfection herein.