What is Data Mesh?
By Varun Kumar | @intelia | May 19
In recent years, there has been a shift in how organisations think about their data. Rather than treating it as a monolithic entity, they are starting to see it as a complex system that needs to be managed in a decentralised way. This shift has given rise to a new concept called data mesh.
Data mesh is a framework for organizing data and data engineering in a way that allows for greater agility, scalability, and flexibility. At its core, it is a way of thinking about data as a product that needs to be designed, built, and maintained with the same care and attention as any other product.
The basic idea behind data mesh is to break down the traditional monolithic approach to data engineering into smaller, more manageable pieces. Instead of having a central team responsible for all aspects of data management, data mesh advocates for creating small, autonomous teams that are responsible for specific data domains.
These teams are cross-functional and are made up of people with a variety of skills, including data engineers, data scientists, domain experts, and product owners. They are empowered to make decisions about their data domain and are responsible for everything from data quality to data modelling to data governance.
The diagram shows the following components:
- Data sources: These are the raw data used to create data products.
- Data infrastructure: This is the hardware and software used to store, process, and analyse data.
- Data catalog: These are the processed and analysed data products that are made available to users.
- Data consumers: These are the users who consume data products via self-service subscriptions.
- Data governance: This is the set of policies and procedures that govern the use of data.
Benefits of using a data mesh architecture:
- Agility: The data mesh architecture is designed to be agile, allowing for rapid development and deployment of new data products. Because each team is responsible for a specific data domain, they can move quickly and make changes as needed without having to go through a centralised team. This means that organisations can respond more quickly to changing business needs and can iterate on their data products more quickly.
- Complexity: Another benefit of data mesh is that it can help to reduce the complexity of data engineering. By breaking down data management into smaller, more manageable pieces, it becomes easier to understand and control the data pipeline. This can help to reduce the risk of errors and can make it easier to identify and fix problems when they occur.
- Innovation: The data mesh architecture encourages innovation, as domain teams are empowered to develop and deploy new data products to meet their needs.
- Sharing: The data mesh architecture makes it easier to share data across domains, as data products are published with well-defined data contracts.
- Governance: The data mesh architecture can be used to implement a strong data governance framework, ensuring that data is used in a responsible and compliant manner.
- Data Quality: Finally, data mesh can help to improve the quality of data. By creating small, autonomous teams that are responsible for specific data domains, organisations can ensure that the people working on the data have a deep understanding of the domain and the data itself. This can lead to better data modelling, better data quality, and ultimately, better insights.
Challenges of using a data mesh architecture:
- Complexity: The data mesh architecture can be complex to implement and manage.
- Skills: The data mesh architecture requires a different set of skills than traditional data warehousing architectures.
- Culture: The data mesh architecture requires a change in culture, as it requires domain teams to take ownership of their data.
Industry use case – Retail
A data mesh architecture could be applied to an organisation in the retail industry in a number of ways. For example, it could be used to:
- Break down data silos: In a retail organisation, data is often stored in silos, with different teams responsible for different parts of the business. A data mesh architecture could break down these silos by creating small, autonomous teams that are responsible for specific data domains, such as inventory, sales, and customer data. This would enable teams to work more collaboratively and share data more easily, leading to better insights and more informed decision-making.
- Improve customer insights: By decentralising data ownership and management, a data mesh architecture can help retailers to gain a deeper understanding of their customers. This can be done by allowing domain teams to collect and analyse data from a variety of sources, such as point-of-sale data, customer surveys, and social media data.
- Enable faster iteration: In the retail industry, it’s important to be able to respond quickly to changing market conditions and customer needs. A data mesh architecture could enable faster iteration by giving teams the autonomy to make decisions about their data domain and make changes as needed. This would allow the organisation to respond more quickly to changes in the market and make more informed decisions about inventory, sales, and marketing.
- Personalise the customer experience: With a better understanding of their customers, retailers can use a data mesh architecture to personalise the customer experience. This can be done by recommending products that customers are likely to be interested in, providing targeted discounts, and sending personalised marketing messages.
- Optimise supply chain management: A data mesh architecture can also be used to optimize supply chain management. By collecting and analysing data from a variety of sources, retailers can gain insights into demand, inventory levels, and transportation costs. This information can be used to make better decisions about where to source products, how much inventory to hold, and how to ship products to customers.
- Streamline data governance: Retail organisations need to ensure that their data is managed in a compliant and secure way. A data mesh architecture could help to streamline data governance by creating a set of shared services that each team can use to manage their data, such as data discovery, data lineage, and data governance. This would ensure that data is managed consistently across the organisation and that compliance requirements are met.
- Foster innovation: A data mesh architecture can also help retailers to foster innovation. By decentralising data ownership and management, it allows domain teams to experiment with new data-driven products and services. This can help retailers to stay ahead of the competition and deliver a better customer experience.
Overall, a data mesh architecture can be a powerful tool for retailers who want to gain a deeper understanding of their customers, personalise the customer experience, optimize supply chain management, and foster innovation.