For a few years now, big data has been one of the hottest types of business technology around. The high-visibility big data successes like Amazon, Facebook, and Netflix have sent everyone from Fortune 1000 businesses to SMEs of every kind scrambling to climb aboard the big data train. In the process, they’ve created a labor market that’s short on experts, and that has seen some of the strongest wage growth of any sector.
What they haven’t produced much of, though, is successful big data projects. In the rush to get up and running, many companies have hit significant stumbling blocks along the way. The situation is so grim that analysts have pegged the big data project failure rate as high as 85%. The biggest underlying problem seems to be that businesses aren’t building the kind of big data teams they need to get the results they want. To help correct that, here is an overview of four critical big data job positions and why they matter for businesses trying to transform into data-driven organizations.
Data Engineer
Of all of the big data careers out there, you can make a convincing argument that data engineers are the most critical among them. The reason is that they serve as the caretakers of all of a company’s data assets. They gather, sort, organize and store every piece of data a business collects “ often from multiple systems and modalities. They’re critical to overall big data success because they make sure that the right data gets into the right hands, so it can be processed by the data scientists and analysts that have the tools and knowledge to make the best use of it. In short, they feed the big data machine and keep it healthy, and without them, no big data initiative can succeed.
Systems Architect
If data engineers are the caretakers of the big data world, then system architects are its master builders. Their primary purpose is to design and build the technology infrastructure needed to support big data initiatives and projects. In many larger organizations, system architects will already be on staff building the larger IT infrastructure needed for day-to-day operations. Within a big data team, they have to understand the nature, size, and complexity of the data the business needs to work with so they can build a system robust and efficient enough to deal with it. When there are existing systems already in use, they find the best ways to integrate them into one, coherent big data platform.
Business Analyst
If there is one position that’s the most pivotal one for a business starting a big data program, it’s that of a business analyst. They serve as a critical bridge that keeps a company’s big data projects aligned with company goals. To do it, they must be true cross-functional employees that understand both the technical and scientific side of big data and the specifics of the business and industry they’re working within. They also help to keep communication between big data professionals and the businesses they’re serving clear, concise, and comprehensible to all. Like the unheralded team of linguistic experts that make the United Nations work “ they’re a vital go-between that’s required for a big data project to succeed.
Data Scientist
As it turns out, the most well-known position in the big data industry is also the least well understood. A data scientist, as their title suggests, is the person who applies their skills to extract meaningful insights from a data set. They build algorithms and design machine learning models to create a data analysis and visualization system the whole organization can turn to for answers. To do it, they have to be part mathematician, part programmer, and part artist. It’s a challenging and complex job “ and is often the biggest failure point for businesses new to big data.
The reason is that they expect data scientists to fill all of the roles needed for a big data project to succeed. In reality, they should be the last person brought in, after the positions above have already started (or even finished) their work. It is only when the infrastructure and communication apparatus is in place that a data scientist can do their job. Otherwise, it’s a bit like asking a mechanic to fix a car in the middle of a desert with no tools in sight (and you can imagine how well that would go).
A Big Data Team That Works
A business that approaches building a big data team with a good understanding of the needed players stands a good chance of success. Those that don’t end up as part of sad statistics quoted in articles like this one. The bottom line is that there are a million ways that businesses can get big data projects wrong, but with the right personnel in place, they can get the most out of the effort. Now that it’s been made clear who those people are and what they do, there’s no excuse for businesses to keep botching their forays into big data.
