Analyzing Big Data Will Require More Statistics

I keep saying that the sexy job in the next 10 years will be statisticians.

Hal Varian, Ph.D., Chief Economist at Google

Big Data (Volume, Velocity, & Variety) represents a paradigm shift in approach and methodology for certain applications, yet not in statistical thinking or statistical assumptions. These, we will need in abundance. First, we will expose the growing myth that Big Data implies complete information. At its best it addresses information gaps; otherwise, Big Data can be redundant. Second, we will clarify the value proposition of statistical tools in making inferences from high-volume data. For many business applications, the foremost challenge with Big Data is reducing it to an analyzable size using statistical techniques that retain the information.

Our global community has a low statistical literacy, much lower than its mathematics literacy. Among the business press and Big Data vendors, there are many who are portraying Big Data as complete information. WRONG. This embodies the resurgence of an ancient data myth: larger datasets and censuses always provide more accurate and reliable results than smaller (statistical) samples. WRONG. Congress is still struggling with this one, every ten years, as they keep asking why it is necessary to augment the U.S. Census with a sample.

Solving problems with complete information is appealing to those who want to think only in a deterministic manner and make definitive proclamations. This mindset foregoes accepting uncertainty with the numbers and the benefits associated with statistics. However, there are four common sources of uncertainty with the numbers. The first comes from using one group of data, such as the past, to infer about another group, such as the future. A second source of uncertainty comes from missing observations and a third source comes from measuring the observations (measurement error). The fourth source of uncertainty occurs when we lack a variable we need and must make do with a surrogate variable(s). These surrogate variables do not contain the same information, creating error. Hence, high volume does not complete the information, so we can not bypass statistics.

The value proposition of statistics applied to Big Data is extensive, going well beyond the above three statistics problems. To clarify how data analysis will always involve statistical thinking, statistical techniques, and statistical assumptions, let us take a closer look. For all data, quants apply three tool boxes: mathematics, statistics, and algorithms. Mathematical tools, which are coated in logic and wrapped around algorithms, address complete numbers. Statistical tools, coated in logic and mathematics and wrapped around algorithms, address incomplete information. The coatings provide rigor. For complete numbers, we can deduce; for incomplete information, we must infer. Algorithmic tools (logic, heuristics, and optimization) work in both the complete and incomplete domains, yet they work differently. The three tool boxes have strong interdependencies and we refer to quants as those professionals, who employ all three.

We do not want unrefined high volume data! We want the decision-affecting information that it contains. After reducing the data, we can employ proven mathematical, statistical, and algorithmic tools. Furthermore, the analytics problem is only a part of the business problem to be solved within certain constraints: Timeliness, Client Expectation, Accuracy, Reliability, and Cost. The bulk of Big Data can, in fact, turn out to be an impedimentand this grows more apparent as the analytics problems become more complex. Reducing the variables and observations, while retaining the wanted information, saves Time, improves Accuracy and Reliability, and lowers Costs.

Conclusion

It is going to take quants to deliver the real promises of Big Data. An understanding of statistics is necessary to access how to analyze Big Data and to properly lead and organize the analytics resources handling it. As Deming said, The nonstatistician cannot always recognize a statistical problem when he sees one. We should expect depictions of Big Data, which are void of an understanding of statistics.

Even if we can get by without addressing statistical errors or reducing the data, we still need statistical thinking, statistical assumptions, and statistical techniques. We need to combine our knowledge about the business problem with a mastery of techniques from all three tool boxes.

Our generation of quants spent our time wanting more data. The next generation will fully experience wanting less.

We sure could use Deming, right now.