Data mesh is the nudge that puts us on a new trajectory in how we approach data: how we imagine data, how we capture and share it, and how we create value from it, at scale and in the field of analytics and AI Zhamak Dehghani
There is a lot of excitement (and information) around Data Mesh at present; What is it? How can I benefit? How do I get data meshed and should I?
Well, they are very good questions. Just because it appears to be the next exciting development in data and will solve all my problems. Will it really?
Weve been through a few interesting next silver bullet developments in data over the years.
Data science (and Big Data).its not that mysterious black box to generate magical opportunities to solve all your problems, though I think some Data Scientists would still like you to believe that. It does however, when done correctly provide insights into the opportunities available to operations and serve your customers better.
The pillars of the data mesh paradigm
Data Catalog.having a view of where all you data is and how it is used is critical. Particularly with compliance responsibilities and helping your Data Scientists find the most appropriate data more easily, is exceptionally useful. Fancy going to Amazon and not having any search or categorisation. How would you find those new Bluetooth headphones you wantyoud probably give up after a while of sifting through loads of products like a jumble sale. You cant implement a Data Catalog and think your Data Management actions are complete or thats all you need to do to obtain great insights!
Then today there is Data Mesh & Data Fabric. I originally wrote about the benefits of Big Data Fabric in Aug 2017. Not much has changed in the concept since then. However, the technology to enable Fabrics and Meshes have evolved to a point where its now a realistic benefit.
The Basics
A data fabric and a data mesh both provide an architecture to access data across multiple technologies and platforms, a data fabric is technology-centric, while a data mesh focuses on organisational change.
Data fabric connects data from multiple sources and prepares it in a way in which you a business user or a data professional can access and analyse it easily. Think about it like putting a big blanket around your architecture to easily access what you need from the outside of the blanket.
According to Gartner, data fabric is a design concept that serves as an integrated layer (fabric) of data and connecting processes. It is a composable, flexible and scalable way to maximize the value of data in an organization. Its not one tool or process, rather an emerging design concept that gives a framework to think about how to stack existing tools, resources, and processes.
Whereas, the data mesh paradigm stands for decentralised and domain-specific data ownership that is easily discoverable and ready for consumption for everyone in the organisation via a Marketplace.
Data mesh moves away from the concept of storing, transforming, and processing analytical data centrally. Instead, it advocates that each business domain is responsible for hosting, preparing, and serving their data to their own domain and larger audience. Dont worry you dont need to be highly technical to be, or remain a Data Owner. Data Ownership sits firmly under the business responsibilities and Data Mesh reinforces that.
Several companies have publicly presented their data mesh strategy here are just a few:
- Intuits breakdown of their data mesh strategy.
- JPMorgan Chase shared their experiences on how to get started with data mesh implementation.
- HSBC uses data mesh in their data strategy
- Learn about data mesh deployments from the awesome companies listed here.
Choosing a fabric or a mesh essentially boils down to the problem your organisation is dealing with.
Is A Data Mesh Right For Me?
Here I take a look at the pros, cons and considerations of a Data Mesh.
Pros
Reinforces the Data Ownership model as Data Product Owners report into Data Owners in the Business, not IT. Data Owners drive Data products within a Data Mesh. IT or Technical experts serve the needs of the Product Owner and the business
Domain teams are in the driving seat of their own Data Products to explore more scenarios
Data Culture matures as each domain is invested in the quality of their data as it directly impacts their domains analytical and insights generation abilities
It is easy to identify what data is available for reuse in the Marketplace
Cons
If you dont have a strong Data Ownership culture, itll be very difficult for a Product Owner to operate without the Data Owners direction
Having a Data Mesh doesnt negate the need for robust Data Management, infact it necessitates the need as more critical
A central team developing Data Products could lead to a backlog, whereas distributed Domain teams could lead to under utilised resources
That investment you have made in creating data products, yet they are not used and colleagues still retain the use of their spreadsheets of siloed data. If the Data Products are not used your investment is a *White Elephant. This is why this must be a culture change not a build it and they will come.they wont!
If sufficient data product maintenance is not in place the data product will become out of date. Another reason why they wont come to your party without sound reassurance of effect maintenance
The pull for a Data Mesh approach needs to come predominantly from the business. The IT approach of if we build it they will come, wont work with the culture change needed for these new business responsibilities and ways of working
Things to Consider
Data Product maintenance workload, should not be underestimated
Separate capabilities and technology for Data Management is required as Data Mesh does not provide Data Management for good solid foundations for your data
How will you measure adoption and usage, and track value of the utilising of the domains data products by other teams and attribute success back to the data domain owning the data product to demonstrate success?
Will your take a central or Domain developed Data Product development team and how will the skills and capabilities available dictate this in the short term?
Security around access to the Marketplace and Data Products needs to be carefully defined with the correct guardrails for what each function will be doing with the data for example utilise adaptive governance with those in a sandpit testing out innovation needing less governance than those creating external facing data solutions
Does the business understand data mesh and data products?
Is the Data Owner ready to run a pod team to create data products?
How to stop business focusing on routine and start considering the art of the possible
“Imagination will often carry us to worlds that never were. But without it we go nowhere” Carl Sagan
Decentralise your domains slowly and carefully with a good level of change management support and data literacy.remember to bring people with you
The need for a data catalog to show what data is available
A data marketplace is needed to share data and the place where you would go to access the Data Products This needs to be a virtual copy of the data product otherwise data is duplicated and cloud storage costs will spiral my understanding is that the *hyperscales are still working on a solution for this, with Snowflake being the most mature in their thinking for a solution
Assessing the maturity and level understanding of the business to accept new ways of working with data, will considerably affect whether you are ready for Data Mesh. Take your time , there is no rush. Get good solid foundations in place first.
Data Mesh is not a silver bullet!
Data Science / Data Catalog / Data Mesh.they all have their place and benefits, none are that magic wand solving all your data challenges. The foundations for good quality, accurate, consistently understood and utilised data with the appropriate level of Governance and security remain critical to get in place before any other activities. I include transformation and migration projects in that bucket of, before any other activities.
Getting data right takes determination, commitment, investment in people, technology and time and mostly leadership to bring your colleagues with you on the journey to Get Data Right and utilise it to generate value.
