Centralised architectures such as data warehouses and data lakes have been the default structures for data offices for nearly a decade. By bringing all enterprise data together in one place, these systems provide a solid foundation for analytics and other data-driven use cases. But, as many organisations have experienced, the use of these systems can cause a range of limitations and bottlenecks.
That has left a lot of data decision-makers asking the same question: is there an alternative that can help drive value from our data quickly and securely without creating huge, centralised workloads? Danilo Sato, head of data & AI services UK and Europe at Thoughtworks says there is: data mesh.
Data mesh is a decentralised architecture approach where, instead of a single monolithic data platform, data is managed as distinct, domain-oriented products by the teams closest to it. Each domain can self-serve and build its own products, while the foundation of the mesh ensures governance and interoperability between those products so that they can be utilised by other domains across the business.
DataIQ, caught up with Danilo to hear about the use cases for data mesh, what it can do for today’s organisations and why adoption is growing.
DataIQ: Why are organisations investing in a data mesh approach? What’s in it for them?
Danilo: Data mesh helps solve a lot of the biggest data-related challenges faced by organisations today, but its biggest benefits lie in how it helps teams completely transform the way they think about and work with data.
Take reducing bottlenecks, for example: The decentralised structure of data mesh enables teams to self-serve when they want to operationalise new data, rather than every team depending on a single centralised team to respond to their requests. That is a big benefit on its own but consider what removing those bottlenecks really means for a business and its culture.
With everyone empowered to help themselves to the data products they need, time to insight is reduced, barriers to innovation are removed and teams naturally start to share more data between them because of increased accessibility. For a lot of organisations, that isn’t just a utopian view of how they’d like to work with data, it’s the data culture they need to keep up in increasingly fast-paced markets.
As an example, Thoughtworks has been working with a broadcast and media entertainment client to help them make the most of several new digital opportunities. As streaming becomes the dominant way that people consume content, the key to success is understanding customer behaviour. By implementing a data mesh, we’ve created a foundation where teams across the organisation can seamlessly share data – simultaneously using it to improve segmentation, drive advertising revenue, increase viewer engagement and deliver personalised content suggestions.
DIQ: What are the main benefits of data mesh over centralised data architectures?
Danilo: As well as cutting time to insight and value by removing those bottlenecks, data mesh can help teams significantly improve data quality. Within a data mesh, data is owned and controlled by the domain teams that know it most intimately.
There’s no long wait time, no need for ‘shadow data’ operations and no siloed knowledge.
Their expertise helps contextualise data, because as we’ve learned over decades of software creation, quality needs to be built in from the start, not inspected or added on at the end. With data mesh, domains are intrinsically incentivised to maintain high data quality, because they’re the ones that will ultimately use and share it across the organisation. If they cut corners, their outcomes and outputs will suffer.
Decentralisation helps disseminate data skills and expertise across the enterprise. Rather than relying on a single team of data experts, you end up with data experts everywhere, which helps teams continuously spot and create new high-value use cases for their data. You not only heighten data literacy within the business, but you also bring business literacy to the data experts.
The end result is a truly data-driven organisation where every team is empowered to get whatever they want from their data, whenever they want it. There’s no long wait time, no need for ‘shadow data’ operations and no siloed knowledge.
DIQ: What kinds of business is data mesh for?
Danilo: It’s fair to say that data mesh probably isn’t a suitable architecture approach for smaller businesses that plan on staying small. But, for everyone else, data mesh can be applied across businesses with high growth aspirations, or organisations with multiple divisions, territories and business units.
The four core principles of data mesh make it possible for any organisation to scale their analytics capabilities quickly and efficiently. Even if you start with a single data product, you might not immediately realise all the network effects and benefits of data mesh, but you can adopt the principles and evolve with them in mind – creating a scalable data foundation and organically embedding a strong data culture across your growing enterprise.
Data mesh can be applied across businesses with high growth aspirations, or organisations with multiple divisions, territories and business units.
DIQ: And how about data maturity? Is data mesh primarily for teams with mature data strategies?
Danilo: Data mesh certainly isn’t only for organisations with mature data strategies, but it will likely appeal to organisations in different ways depending on their previous experiences and data-driven initiatives. For example, if an organisation is feeling the negative symptoms that stem from centralisation – such as bottlenecks, ownership concerns, not receiving the speed of data access that is wanted or persistent data quality issues – that would be a big driver to consider data mesh.
Data mesh certainly isn’t only for organisations with mature data strategies
If an organisation doesn’t have a mature data strategy, data mesh is a great place to start. It gives organisations a way to embrace data ownership boundaries early on in their journey and avoid having to incur and overcome the challenges associated with traditional centralised approaches. Data mesh forces organisations to think carefully about ownership from the get-go, helping teams develop strong internal data structures and make concepts like product-thinking an embedded part of their culture.
DIQ: What investment is required to get started with data mesh?
Danilo: There is an activation cost to start building the mesh platform. That can include a mixture of infrastructure costs and the cost to build the foundational platform itself, plus the cost of acquiring any external data products that you require to support your own.
But organisations should try not to get too hung up on costs. Instead of focusing on the activation cost, we encourage teams to consider the cost of not modernising their architecture. To succeed in the future, stay competitive and rapidly convert data into business value, organisations need both an agile data foundation and the right culture –and we believe that data mesh can enable both.
Click here to find out more about Data Mesh as a solution and to contact the Thoughtworks team.
Click here to join DataIQ to become a part of the leading data community, develop the data capabilities of your team and organisation and connect with peers at engaging roundtables and events.
Thank you for your input
Thank you for your feedback
DataIQ is a trading name of IQ Data Group Limited
10 York Road, London, SE1 7ND
Phone: +44 020 3821 5665
Registered in England: 9900834
Copyright © IQ Data Group Limited 2024