Over the past few years, I have repeatedly written about the endless benefits of big data. However, I don’t write enough about the problems that transpire when big data projects are improperly managed.
A few years ago, a colleague of mine operated a company that was trying to develop a variety of AI applications for startups in the Bay Area. Due to the sophisticated nature of the problems that he was trying to solve, he relied extensively on big data. He and his team members were very knowledgeable data scientists and web developers, so they were certain that the solutions they developed would be highly useful to their future clients.
They were in for a rude awakening when they finally pitched to a growing company in San Francisco. They had a presentation with the company and showed a demonstration of the final application. Unfortunately, the company swiftly rejected the proposal. The harsh reply indicated that other potential clients would probably be equally disinterested.
My colleague’s team reached a disheartening conclusion. They developed a poorly thought out big data application that nobody would purchase. Five months of their lives and over $10,000 was wasted.
What went wrong? The biggest mistake they made was delegating all engineering and design decisions to backend developers and data scientists. While there were many knowledgeable big data experts on their team, they didn’t have a solid background in marketing or UX, which are equally important elements of the pipeline. In this cases, they were working with companies in the equity financing sector. They failed to understand the use of virtual data rooms, 32-bit encryption and other big data solutions for managing mergers and acquisitions and other solutions the client needed.
Nobody stopped to consider the issues from the end customer s point of view. Here are some of the common data pipeline development mistakes that can ruin the outcome of the project.
Neglecting to use compression
Some applications require several hundred gigabytes of data. If your data isn’t synthesized properly, you can encounter any number of challenges ranging from processing lags to system crashes.
One of the best ways to address this is by compressing all files during the initial design phases. While some of the structured data sets that you incorporate into the project will be smaller than others, they all collectively add to the volume of data. Compressing every file and data source streamlines the process.
Making faulty assumptions about the customers’ needs
My colleague assumed that he knew what his customers would need. His assumptions proved to be incorrect, which is why his project failed.
Sophisticated knowledge of big data won’t make up for a faulty understanding of your customers. Conducting customer focus groups and surveys is crucial for any project. It will save you from spending thousands of man-hours on features that don’t align with their needs.
Failing to structure data pipelines into smaller tasks
Transferring massive volumes of data can be impossible. You need to break your project down into much smaller tasks to accomplish this.
Alois Barreras, a big data developer for Astronomer, provides a detailed overview of this challenge.
Writing data pipelines that scale is really hard. Just because a data pipeline moves 10,000 rows of data from Mongo to Redshift with no problem does not mean the same thing will happen with 50 million rows (ever seen an ENOMEM error?). A common data pipeline used by our clients is Mongo to Redshift. Writing one giant pipeline ”one that pulls in data from Mongo, transforms it and sends it to Redshift ”restricts that pipeline to only being useful to someone who wants to send data from Mongo to Redshift. At Astronomer, we identify the individual parts of a data pipeline and break them up into distinct tasks (check out our GitHub repo for the list and source code for all of our integrations).
You can minimize the burden on your server by understanding the bandwidth and logistical limitations of each Hadoop tool, server and another element in your pipeline. However, the entire process will be much more manageable if you divide it up into smaller tasks.
