I am sure most of you must be familiar with the idea of DevOps, a series of slick processes applied to smoothen development with IT operations replacing the other proven yet slower methods used by IT teams to come up with interesting enterprise applications. And there is no doubt in the fact that DevOps is now becoming business as usual’ at most large organisations. After all, tech giants like Facebook, Google would not build apps any other way. It may even interest you to know that the analyst firm Gartner has become cautiously optimistic about the new way of working. According to Gartner, DevOps will evolve from a niche to a mainstream strategy employed by 25 percent of Global 2000 organisations.
Now apart from DevOps, if I ask to list out the top trends that are shaping the enterprise data centre today than the technologies would include cloud computing, containers and virtualisation, microservices, machine learning and data science, flash memory, edge computing, NVMe and so more. Over the past few years, these technologies have been more than a nerdy concept for organisations pushing digital transformation. Recently, a new trend named DataOps came into existence to increase traction within large enterprises.
About DataOps
DataOps is generally defined as a cousin of DevOps, a data management method that emphasises communication, collaboration, integration, automation, and measurement of cooperation between data engineers, data scientists and other data professionals. But the reality feels a million miles away from this; the method comprises of four layers or four skill sets:
- Information consumers– Make business decisions, e.g. CFOs, CIOs, etc
- Business Analysts– Understand the data and create business information via data models, e.g. Business Intelligence/Business Warehouse consultants, report writers and now, Data Scientists
- Platform operations– understand how to manage the data platform, e.g. Database Administrators and now, DataOps professionals
- Infrastructure Operations – understand how to manage the IT Infrastructure housing the data, e.g. managers of the hardware and Operating Systems, Sysadmin’s, and now DataOps
Its benefits include:
- Faster time to value when working with data, thanks to streamlined communication across the team.
- Problems can be identified more quickly especially the ones which could lead to errors or delays. By detecting issues earlier, before they snowball into larger problems, you can fix them more easily.
- Stronger security. When the security experts are in close contact with everyone else on the data team, securing data becomes easier.
- The method provides better use of the staff time. I mean as soon as the communication gets streamlined, your experts can spend more time putting their expertise to work and less time trying to communicate information to each other.
The Journey- DevOps to DataOps
The democratisation of analytics, in particular, is providing individuals easy access to cutting-edge visualisation, data modelling, machine learning and statistics. According to Christian, CEO of Tableau, DataOps turns out to be a tremendous opportunity to help people answer questions, solve problems and generate meaning from data in a way that has never before been possible. And we believe there’s an opportunity to put that power in the hands of a much broader population of people.
By implementing built-for-purpose database engines, one can radically improve performance and accessibility of large quantities of data at unprecedented velocities. In fact, Google has made its massive Bigtable database (the same one that powers Google search, Maps, YouTube, Gmail, etc.) available to everyone in an extremely scalable NoSQL database service accessible through the industry-standard, open-source ApacheHBase API.
Together these trends end up creating pressure from both ends of the stack. What I mean is from the top of the stack, more users want access to more data in more combinations and from the bottom, more data is available than ever before.
Bi Is Not DataOps
Now I have come across many companies making use of BI to review and analyse company data, but it was never a 24/7 job and of course, data never sleeps. Despite knowing the fact, BI professionals haven’t developed the network operations centre-type. Always keep this in mind, to introduce new roles to entrenched companies is not an easy thing, but adoption of new practices never is. And to keep up with data, businesses are going to have to reframe the way they think about monitoring and develop new job roles to accommodate this change.