• Skip to main content
  • Skip to secondary menu
  • Skip to primary sidebar
  • Skip to footer
  • Articles
  • News
  • Events
  • Advertize
  • Jobs
  • Courses
  • Contact
  • (0)
  • LoginRegister
    • Facebook
    • LinkedIn
    • RSS
      Articles
      News
      Events
      Job Posts
    • Twitter
Datafloq

Datafloq

Data and Technology Insights

  • Categories
    • Big Data
    • Blockchain
    • Cloud
    • Internet Of Things
    • Metaverse
    • Robotics
    • Cybersecurity
    • Startups
    • Strategy
    • Technical
  • Big Data
  • Blockchain
  • Cloud
  • Metaverse
  • Internet Of Things
  • Robotics
  • Cybersecurity
  • Startups
  • Strategy
  • Technical

Impress With Data Mesh

Liz Henderson / 7 min read.
May 2, 2022
Datafloq AI Score
×

Datafloq AI Score: 84.33

Datafloq enables anyone to contribute articles, but we value high-quality content. This means that we do not accept SEO link building content, spammy articles, clickbait, articles written by bots and especially not misinformation. Therefore, we have developed an AI, built using multiple built open-source and proprietary tools to instantly define whether an article is written by a human or a bot and determine the level of bias, objectivity, whether it is fact-based or not, sentiment and overall quality.

Articles published on Datafloq need to have a minimum AI score of 60% and we provide this graph to give more detailed information on how we rate this article. Please note that this is a work in progress and if you have any suggestions, feel free to contact us.

floq.to/xnX4n

Data mesh is the nudge that puts us on a new trajectory in how we approach data: how we imagine data, how we capture and share it, and how we create value from it, at scale and in the field of analytics and AI Zhamak Dehghani

There is a lot of excitement (and information) around Data Mesh at present; What is it? How can I benefit? How do I get data meshed and should I?

Well, they are very good questions. Just because it appears to be the next exciting development in data and will solve all my problems. Will it really?

Weve been through a few interesting next silver bullet developments in data over the years.

Data science (and Big Data).its not that mysterious black box to generate magical opportunities to solve all your problems, though I think some Data Scientists would still like you to believe that. It does however, when done correctly provide insights into the opportunities available to operations and serve your customers better.

The pillars of the data mesh paradigm

Data Catalog.having a view of where all you data is and how it is used is critical. Particularly with compliance responsibilities and helping your Data Scientists find the most appropriate data more easily, is exceptionally useful. Fancy going to Amazon and not having any search or categorisation. How would you find those new Bluetooth headphones you wantyoud probably give up after a while of sifting through loads of products like a jumble sale. You cant implement a Data Catalog and think your Data Management actions are complete or thats all you need to do to obtain great insights!

Then today there is Data Mesh & Data Fabric. I originally wrote about the benefits of Big Data Fabric in Aug 2017. Not much has changed in the concept since then. However, the technology to enable Fabrics and Meshes have evolved to a point where its now a realistic benefit.

The Basics

A data fabric and a data mesh both provide an architecture to access data across multiple technologies and platforms, a data fabric is technology-centric, while a data mesh focuses on organisational change.

Data fabric connects data from multiple sources and prepares it in a way in which you a business user or a data professional can access and analyse it easily. Think about it like putting a big blanket around your architecture to easily access what you need from the outside of the blanket.

According to Gartner, data fabric is a design concept that serves as an integrated layer (fabric) of data and connecting processes. It is a composable, flexible and scalable way to maximize the value of data in an organization. Its not one tool or process, rather an emerging design concept that gives a framework to think about how to stack existing tools, resources, and processes.

Whereas, the data mesh paradigm stands for decentralised and domain-specific data ownership that is easily discoverable and ready for consumption for everyone in the organisation via a Marketplace.

Data mesh moves away from the concept of storing, transforming, and processing analytical data centrally. Instead, it advocates that each business domain is responsible for hosting, preparing, and serving their data to their own domain and larger audience. Dont worry you dont need to be highly technical to be, or remain a Data Owner. Data Ownership sits firmly under the business responsibilities and Data Mesh reinforces that.

Several companies have publicly presented their data mesh strategy here are just a few:

  • Intuits breakdown of their data mesh strategy.
  • JPMorgan Chase shared their experiences on how to get started with data mesh implementation.
  • HSBC uses data mesh in their data strategy
  • Learn about data mesh deployments from the awesome companies listed here.

Choosing a fabric or a mesh essentially boils down to the problem your organisation is dealing with.

Is A Data Mesh Right For Me?

Here I take a look at the pros, cons and considerations of a Data Mesh.

Pros

Reinforces the Data Ownership model as Data Product Owners report into Data Owners in the Business, not IT. Data Owners drive Data products within a Data Mesh. IT or Technical experts serve the needs of the Product Owner and the business

Domain teams are in the driving seat of their own Data Products to explore more scenarios

Data Culture matures as each domain is invested in the quality of their data as it directly impacts their domains analytical and insights generation abilities

It is easy to identify what data is available for reuse in the Marketplace

Cons

If you dont have a strong Data Ownership culture, itll be very difficult for a Product Owner to operate without the Data Owners direction


Interested in what the future will bring? Download our 2023 Technology Trends eBook for free.

Consent

Having a Data Mesh doesnt negate the need for robust Data Management, infact it necessitates the need as more critical

A central team developing Data Products could lead to a backlog, whereas distributed Domain teams could lead to under utilised resources

That investment you have made in creating data products, yet they are not used and colleagues still retain the use of their spreadsheets of siloed data. If the Data Products are not used your investment is a *White Elephant. This is why this must be a culture change not a build it and they will come.they wont!

If sufficient data product maintenance is not in place the data product will become out of date. Another reason why they wont come to your party without sound reassurance of effect maintenance

The pull for a Data Mesh approach needs to come predominantly from the business. The IT approach of if we build it they will come, wont work with the culture change needed for these new business responsibilities and ways of working

Things to Consider

Data Product maintenance workload, should not be underestimated

Separate capabilities and technology for Data Management is required as Data Mesh does not provide Data Management for good solid foundations for your data

How will you measure adoption and usage, and track value of the utilising of the domains data products by other teams and attribute success back to the data domain owning the data product to demonstrate success?

Will your take a central or Domain developed Data Product development team and how will the skills and capabilities available dictate this in the short term?

Security around access to the Marketplace and Data Products needs to be carefully defined with the correct guardrails for what each function will be doing with the data for example utilise adaptive governance with those in a sandpit testing out innovation needing less governance than those creating external facing data solutions

Does the business understand data mesh and data products?

Is the Data Owner ready to run a pod team to create data products?

How to stop business focusing on routine and start considering the art of the possible

“Imagination will often carry us to worlds that never were. But without it we go nowhere” Carl Sagan

Decentralise your domains slowly and carefully with a good level of change management support and data literacy.remember to bring people with you

The need for a data catalog to show what data is available

A data marketplace is needed to share data and the place where you would go to access the Data Products This needs to be a virtual copy of the data product otherwise data is duplicated and cloud storage costs will spiral my understanding is that the *hyperscales are still working on a solution for this, with Snowflake being the most mature in their thinking for a solution

Assessing the maturity and level understanding of the business to accept new ways of working with data, will considerably affect whether you are ready for Data Mesh. Take your time , there is no rush. Get good solid foundations in place first.

Data Mesh is not a silver bullet!

Data Science / Data Catalog / Data Mesh.they all have their place and benefits, none are that magic wand solving all your data challenges. The foundations for good quality, accurate, consistently understood and utilised data with the appropriate level of Governance and security remain critical to get in place before any other activities. I include transformation and migration projects in that bucket of, before any other activities.

Getting data right takes determination, commitment, investment in people, technology and time and mostly leadership to bring your colleagues with you on the journey to Get Data Right and utilise it to generate value.

Categories: Big Data
Tags: Big Data, Data, technology

About Liz Henderson

Appealing to and winning hearts and minds for 20+ years, with a proven track record of leading and shaping data strategy, incl. data commercialisation and analytics. Advising on and executing a wide range of corporate compliance initiatives, digital transformations and data migrations, while developing a framework for sustainable expansion and profitable growth from data. When she is not governing data, she enjoys gardening and travelling, has a data blog lizhendersondata.wordpress.com, is a STEM ambassador and a non-executive director #dataqueen

Primary Sidebar

E-mail Newsletter

Sign up to receive email updates daily and to hear what's going on with us!

Publish
AN Article
Submit
a press release
List
AN Event
Create
A Job Post

Related Articles

The Advantages of IT Staff Augmentation Over Traditional Hiring

May 4, 2023 By Mukesh Ram

The State of Digital Asset Management in 2023

May 3, 2023 By pimcoremkt

Test Data Management – Implementation Challenges and Tools Available

May 1, 2023 By yash.mehta262

Related Jobs

  • Software Engineer | South Yorkshire, GB - February 07, 2023
  • Software Engineer with C# .net Investment House | London, GB - February 07, 2023
  • Senior Java Developer | London, GB - February 07, 2023
  • Software Engineer – Growing Digital Media Company | London, GB - February 07, 2023
  • LBG Returners – Senior Data Analyst | Chester Moor, GB - February 07, 2023
More Jobs

Tags

AI Amazon analysis analytics app application Artificial Intelligence BI Big Data business China Cloud Companies company costs crypto Data design development digital engineer environment experience future Google+ government Group health information learning machine learning market mobile news public research security services share skills social social media software strategy technology

Related Events

  • 6th Middle East Banking AI & Analytics Summit 2023 | Riyadh, Saudi Arabia - May 10, 2023
  • Data Science Salon NYC: AI & Machine Learning in Finance & Technology | The Theater Center - December 7, 2022
  • Big Data LDN 2023 | Olympia London - September 20, 2023
More events

Related Online Courses

  • Oracle Cloud Data Management Foundations Workshop
  • Data Science at Scale
  • Statistics with Python
More courses

Footer


Datafloq is the one-stop source for big data, blockchain and artificial intelligence. We offer information, insights and opportunities to drive innovation with emerging technologies.

  • Facebook
  • LinkedIn
  • RSS
  • Twitter

Recent

  • 5 Reasons Why Modern Data Integration Gives You a Competitive Advantage
  • 5 Most Common Database Structures for Small Businesses
  • 6 Ways to Reduce IT Costs Through Observability
  • How is Big Data Analytics Used in Business? These 5 Use Cases Share Valuable Insights
  • How Realistic Are Self-Driving Cars?

Search

Tags

AI Amazon analysis analytics app application Artificial Intelligence BI Big Data business China Cloud Companies company costs crypto Data design development digital engineer environment experience future Google+ government Group health information learning machine learning market mobile news public research security services share skills social social media software strategy technology

Copyright © 2023 Datafloq
HTML Sitemap| Privacy| Terms| Cookies

  • Facebook
  • Twitter
  • LinkedIn
  • WhatsApp

In order to optimize the website and to continuously improve Datafloq, we use cookies. For more information click here.

settings

Dear visitor,
Thank you for visiting Datafloq. If you find our content interesting, please subscribe to our weekly newsletter:

Did you know that you can publish job posts for free on Datafloq? You can start immediately and find the best candidates for free! Click here to get started.

Not Now Subscribe

Thanks for visiting Datafloq
If you enjoyed our content on emerging technologies, why not subscribe to our weekly newsletter to receive the latest news straight into your mailbox?

Subscribe

No thanks

Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.

Necessary Cookies

Strictly Necessary Cookie should be enabled at all times so that we can save your preferences for cookie settings.

If you disable this cookie, we will not be able to save your preferences. This means that every time you visit this website you will need to enable or disable cookies again.

Marketing cookies

This website uses Google Analytics to collect anonymous information such as the number of visitors to the site, and the most popular pages.

Keeping this cookie enabled helps us to improve our website.

Please enable Strictly Necessary Cookies first so that we can save your preferences!