Roundtable: TRUST with Data Mesh

This domain-oriented, decentralised approach to data ownership and architecture can deliver distributed data access, analytics and data governance and transform your organisation.

This live discussion and Q&A featured insights on data mesh adoption from

• Vinay Srihari, Field CTO - Data Cloud, Snowflake
• Siddharth Rajagopal, Principal Architect EMEA-LA, Informatica
• Hemant Waradkar, Global Practice Head - Data & Analytics, Wipro Technologies

These three leaders in the field discuss:

• The most compelling business drivers for the adoption of data mesh
• How to anticipate and overcome the key challenges of data mesh adoption 
• The architectural impact of data mesh and the “three planes” model

Key Drivers for Data Mesh Adoption

Over the past three or four years, factors such as the pandemic and geopolitical instability have forced businesses to seek new business models—and a new, collaborative way of working has emerged.

“The concept of data mesh is catching up across the industry in a very big way,” said Hemant Waradkar, Global Practice Head - Data & Analytics at Wipro Technologies.

“Telcos are collaborating with banks; telcos are collaborating with retailers; retailers are collaborating with manufacturers—intelligent ecosystems are being created… Each one of us can benefit as part of that Intelligent ecosystem.”

“Data is the key asset every organisation is trying to monetise,” Waradkar continued. 

But without the right architecture in place, businesses cannot scale up and derive value from their data.

“Businesses all want to come together to provide a common experience to their customers and suppliers. They also want to ensure that scalability is not restricted to a central team—but is federated towards different departments,” said Siddharth Rajagopal, Principal Architect EMEA-LA at Informatica.

One powerful theme that has emerged over recent years is decentralisation. 

“Every organisation—big and small—is realising that in order to scale, the old model of a central ‘data office’ or a central IT team that is responsible for ‘all things data’… doesn’t really work,” said Vinay Srihari, Field CTO - Data Cloud at Snowflake.

A key tenet of dash mesh is that data is best understood by the teams that work most closely with it. Decentralising control over data harnesses the expertise of people across the whole organisation.

Data mesh is sometimes described as a “buzzword”, but there are “real needs” driving its adoption, said Srihari, including decentalisation, productisation of data, and the need to improve governance.

Biggest Challenges in Data Mesh

Data mesh adoption is—in part—being driven by the buzz surrounding the concept. But to get buy-in from stakeholders, the data mesh project must show a clear return on investment (ROI).

Before an organisation begins with data mesh, it must identify the business outcomes it is trying to achieve, said Wirpro’s Hemant Waradkar.

“The whole organisation need to come together to create a data monetisation aspect, which can—in return—deliver value.”

Data democratisation, too, requires investment from across the whole organisation.

“With the intelligent ecosystem… it is imperative for us to make sure that the data reaches the right stakeholders in the right time,” Waradkar said.

But while data mesh is decentralised by nature, strong leadership can be required to get a project off the ground.

“It helps sometimes if that’s outside the organization because then no one has a vested interest,” said Snowflake’s Vinay Srihari. “It needs a leader—somebody who has decided that this is the path.”

“And so when you have all of those pieces, then things work great,” Srihari continued. “But if any of those have not been thought through—or there are ‘blockers’, that’s where we find that. data mesh implementations are either slow to get going or can collapse.”

Architectural Impact of Data Mesh 

One way of conceiving the architectural impact of data mesh is across three planes:

• Mesh experience plane
• Developer experience plane
• Data infrastructure plane

Let’s consider each of these planes in turn.

Mesh Experience Plane

The mesh experience plane deals with how domains work together and collaborate on data products.

“At the end of the day, every architectural decision that we take has to sustainably provide you with a business outcome,” said Wipro’s Hemant Waradkar. “Unless there is a business outcome and an ROI associated with an investment, nothing will move in any organisation.” 

The mesh experience plane involves creating a dynamic architecture to solve a specific business problem, such as how to improve customer experience.

“Domains, and the way you way you organise domains, need to be very close to business outcomes,” said Informatica’s Siddharth Rajagopal.

“You need to connect the related data, your key data elements, data assets and the data sets for these domains—so that they become a reliable layer on which to build your data products.”

Developer Experience Plane

How do developers create and scale data products? There are several considerations for developers here.

First, bear in mind that the developer experience plan is the middle plane: Developers need to consider the layers above and below. 

“How can [developers] create a developer experience which can serve data products in an agile way for different domains above, and at the same time really consider the performance requirements, the infrastructure requirements and the way the infrastructure is set up?”

Developers also need to ensure that data can be trusted, considering factors such as data quality, data lineage, and master data.

The developer experience plane should also enable templatisation, by building sharable APIs and creating a set of modular libraries that can be reused by other developers.

Another important theme in the developer experience plane is abstraction. 

Developers should create assets that are not overly specific or tied up with the underlying infrastructure plane—but that can be abstracted and reusable irrespective of, for example, moving from on-premises to cloud infrastructure.

Data Infrastructure Plane

The data infrastructure plane concerns how to work with underlying data infrastructure.

“The data infrastructure plane means having data platform that is simple to use,” said Snowflake’s Vinjay Srihari, “Where you can design these reusable components and reusable data flows—like an inject engine and an ingest pipeline—that every organization can make use of to bring in their own data.”

Once a reusable data flow has been implemented, it must be made visible to the whole organisation.

“That’s where something like Informatica data catalogue comes in,” said Srihari. “We use that successfully across the data platform.”

“Where we’ve been successful at Snowflake is in providing that simplicity, that ease-of-use, that consistent look and feel and the ability to automate,” he continued.

“These are really key components for us in the infrastructure plane.”

Data Mesh: People Are Key

Data mesh is a fantastic way to monetise data and improve collaboration within and between organisations.

A prevailing theme across discussions of data mesh is the importance of people and business objectives.

Without a clear business objective driving the implementation of data mesh, the project is unlikely to succeed.

And the decentralised, federated nature of data mesh means that it is vital to secure the investment of people throughout the whole organisation.