Just How Big Can Big Data Get?

At the most significant point of its history, the Ancient Library of Alexandria may have held 500,000 papyrus scrolls, thereby making it a collection of information on par with the data that makes up today’s “visible web.”

This magnificent library suffered a series of fires until its final and complete destruction around 642 AD; at that time, the loss of all those scrolls was as catastrophic as wiping out half of all the servers connected to the internet these days.

It is nearly impossible to imagine how much larger the Library of Alexandria could have grown had it been spared from destruction. Certainly, at some point it would have required physical expansion as well as cataloguing and archiving systems to hold and manage millions of scrolls; t a great extent, this is one of the challenges faced by information technology experts these days as they tackle the issue of Big Data and its constant expansion.

When Big Data Gets Too Big and Cumbersome

With the monumental growth and development that Big Data has experienced in the 21st century, we probably have all the information we need in the enterprise world; even if we consider that more data is needed for future applications, we certainly have the means to obtain it and parse it. While this may sound like an ideal situation, we are also reaching a saturation point.

Modern bar code labels can store approximately 7K of uncompressed data; however, there are very few applications that we have created to take advantage of this capacity. If we stick a smart barcode label to a ceramic coffee cup, we will likely use it to store just a few pieces of information such as pricing, materials, origin, and regional marketplaces; clearly, we could store a lot more data such as marketing strategies and profit margins, but we tend to keep such information separate.

As the aforementioned coffee cup moves through the retail supply chain, lots of information is being generated and collected, but this data production is often conducted without a meaningful business purpose; the information is stored in servers until someone comes around with knowledge about how to use it. We keep investing in Big Data, but we are at the point of diminishing returns.

Cutting Back on Big Data

Thinking back to the Library of Alexandria, it is not difficult to imagine that its editors and librarians did not store each and every piece of papyrus they came across. Scholars working at the Library were the decision makers, insofar as that they chose what should be stored and catalogued.

There needs to be a valid business case for Big Data; otherwise, companies should start thinking about cutting back on such practices until they make actual business sense.

When Big Data Becomes a Risk

Companies that make a business decision to maintain their Big Data practices should realize that there may be some risks involved.

When data sets are made available to remote clients, IT departments must install adequate endpoint protection software. In February 2017, a renowned hacker known as Rasputin was able to breach more than a dozen American universities using SQL injection attacks, and he mainly targeted Big Data servers where research information was stored and made available for remote access.

In the end, we should not be getting too far ahead of ourselves with Big Data. Each application should be designed with specific goals in mind; for example, allowing security guards to count the number of cars in a parking lot at any given time should not be presented as a series of menus. Simply presenting a number or a visual representation should be sufficient.

Furthermore, Big Data dashboards should be coded in a way that they can be scaled; giving end users too much information to evaluate could become a problem in the near future.