How to Close the Big Data Talent Gap at Your Organization

This article is sponsored by Acamar – a staffing agency specializing in emerging Information Management (IM) competencies.

Big data offers many benefits for organizations in all industries, but unfortunately, a lot of companies dont reap these benefits yet. The reason is not that they dont want to start with big data, nor that they dont understand what big data is. The challenge many companies face is attracting the right big data talent.

Big data talent is scarce and what is scarce is expensive. Finding the right data professionals for a big data project remains difficult for a lot of organizations. This does not come as a surprise if we look at the numbers around global big data talent.

A Growing Global Big Data Talent Shortage

Back in 2011, McKinsey already estimated that in 2018 there would be a shortage of 290,000 data scientists in the United States alone. Globally, demand for data scientists is projected to exceed supply by more than 50 percent by 2018. In the UK, the expected shortage is 56,000 data scientists by 2020. Currently, there are over 500,000 big data jobs listed globally.

As it may be clear, demand is skyrocketing and this will only increase in the future. On the other hand, big data master programs have only started to appear in the past five years or so. Currently, there are only a few hundred data science master programs, barely enough to cater for future demand.

The result is scarcity in available big data talent, which is unfortunately not the only challenge for organizations. Most organizations have no idea who to look for. The term Big Data Scientist is over-hyped and almost every business wanting to deal, or dealing, with big data is seeking to hire this illustrative big data scientist while they have no clue what the Big Data scientist should do. To make matters worse, there are many ordinary data analysts out there who have taught themselves R or something else and now call themselves Big Data Scientist,’ to make an extra buck of this hype.

Understanding The Required Big Data Skills

These two challenges, therefore, make it difficult for organizations looking to hire big data talent. Therefore, it is important to know what they need and which skills they should hire. This will, on the one hand, prevent hiring these fake data scientists and, on the other hand, will help you hire a better fit for your business.

Of course, what skills are required all depends on the big data projects being developed, the industry, the type of organization and many other variables. However, as Louis Colombus writes on Forbes, we can distinguish ten skills that are most in demand by organizations. The top three of this list includes Python, Linux and Structured Query Language, with Apache Hadoop currently in position eight.

Of course, this list of skills is constantly changing, especially since we are slowly moving to the algorithmic business as described earlier. It is therefore quite likely that within a few years the list includes important topics such as machine learning, algorithm design or artificial intelligence. As a result the search will become even harder for organizations.

Finding the Right Big Data Talent

Knowing that there is a shortage and that many organizations dont know what to look for, how can organizations close this big data gap and hire the best talent for their projects?

First of all, you can look internally to see which employees show the potential to become a big data professional. Without knowing it, you might have a great big data talent in the making within your organization who, with the right training, can become a top big data scientist.

In case you already have some great big data professionals within your organization, but are looking for more, another option can be to optimize and amplify existing big data talent through a Center of Excellence. Within this Center of Excellence, the existing big data talent is grouped together and works for the entire organization instead of a single department. An additional advantage is that top Centers of Excellence, if executed correctly, can attract other talent that was hard to attract before.

If these two options dont work for you, it is time to start looking externally. Before you hire a recruitment agency such as Acamar to help you find the big data talent, understand why you need big data professionals and what skills they should have. As mentioned, this depends on the big data project and the better you know the exact expertise and technologies you are looking for, the better you will be able to find the right big data professional. In addition, provide critical input on talent needs who, what, when, where, how to better facilitate the talent deployment and better attract and retain the big data professionals.

But having a large pool of data scientists to choose from is not the end game. In the end, it is making sure that the right big data professional works for the right company and works on the right big data project. After all, the manufacturing industry requires a different type of Big Data talent, with different skills, than for example the financial services industry.

Finding the right talent for the right job remains a challenge, but fortunately we also see more and more specialized recruiting agencies such as Acamar who are dedicated to this growing field of finding the best suitable big data talent for your organization.

Finally, if you manage to attract big data professionals, ensure that you treat them well. Restricting big data professionals too much, will very likely result in an empty big data department.  On the other hand, offering them exciting and challenging projects and giving them the freedom to bring solutions to the table you wouldnt have thought of will likely attract more big data talent.