This is a sponsored blog post by Saurav Gupta, Sales Engineer, InterSystems
Financial services organizations are awash with data, and there’s a clear appetite in the sector to make use of it for a wide variety of initiatives, including analytics on real-time transactional data and reducing customer churn. But doing so requires putting the right data management architecture in place. That is rarely easy. Over the years, organizations have tried different ways to deliver consistent views of enterprise data to support their business needs but rapid changes in the demands of what their IT infrastructure and data environments need to deliver, like the implementation of data lakes and data warehouses, mean that challenges still remain.
While data within financial services organizations is often siloed and difficult to access and consume, we are now seeing the emergence of new approaches to data management that can overcome these challenges. Two of the most promising: data fabric and data mesh, are designed to help organisations leverage maximum business value from their data and existing data infrastructure.
There are many similarities between the two approaches. Both allow the data to remain stored in place at the source – a key differentiator over legacy systems that require data to be copied and moved using batch processes.
In addition, both a data fabric and a data mesh connect disparate data and applications, including on-premises, from partners, and in the public cloud, to discover, connect, integrate, transform, analyze, manage, and utilize them. By leveraging these capabilities, both approaches enable the business to meet business goals quickly and efficiently.
Despite the parallels between the two, there are also some important differences to consider here, which highlight why they are complementary rather than interchangeable. With a data fabric, the metadata, governance, and semantics are managed centrally. This structure is more frequently encountered in financial services companies that employ a Chief Data Officer that takes a top-down approach to data management.
The latest iteration, smart data fabrics, build on the data fabric foundation and incorporate a wide range of analytics capabilities, including data exploration, business intelligence, natural language processing, and machine learning directly within the fabric itself. For financial services, this means there is an ability to perform analytics on real-time event and transactional data, without impacting the performance of the transactional system. Organizations can move away from querying on offline or intraday numbers, to making decisions in the moment with real-time insights.
A data mesh, on the other hand, enables local domain teams to own the delivery of data products based on the premise that they are closer to their data and understand it better. It’s supported by an architecture that leverages a domain-oriented, self-serve design, enabling local teams to discover, understand, trust, and use data to inform decisions and initiatives and develop and deploy data products and applications.
One key difference between the two is that a data mesh allows data governance to be defined and managed at the source systems (endpoints), while a data fabric provides an overarching fabric that includes governance, lineage, security, etc., applied and managed centrally, for example, by the CDO. Looking at this in practical terms, a data mesh may be appropriate for situations where there are data sovereignty concerns, whereas a data fabric may be the right approach where the office of the CDO is defining an organizational taxonomy with access privileges.
These points of differentiation highlight the fact that the two approaches are not mutually exclusive – far from it. In fact, when it comes to determining which type of architecture to use, the selection is dependent upon the business use case. If the senior team wants to have an enterprise view of their data assets with enterprise level governance, for example, they will likely choose to implement an enterprise data fabric. If the organization wants to empower certain trusted parts of the enterprise with the flexibility to create and manage their own applications to speed innovation and digital transformation initiatives, or if data sovereignty issues are of concern, a data mesh may be an appropriate component of their overall architecture.
However, it’s equally true that, in the right circumstances, the two approaches can, and often do, work together positively to achieve positive outcomes. As one of our major financial services customers puts it: “Fabric and mesh share the same goal of easy access to data, and under the right circumstances can in fact be complementary approaches.”
The reality is that data fabric architectures can co-exist with data mesh initiatives where it makes sense, such as in large organizations that must manage campaign data locally within regions.
One example where a data fabric and a data mesh work simultaneously can be seen in the demands of a large multinational wealth management firm with customer 360 initiatives.
In this use case, the company’s overall data strategy is managed centrally (data fabric), but sovereignty issues over data retention and processing are present in certain countries where local marketing campaigns are being executed. Allied to this, there is specific local knowledge of the customers in the regions, which informs variations in local campaign management. These variations are dealt with by the regional, country, or local IT teams (data mesh).
These kinds of practical examples of how data mesh and data fabric can work together to deliver tangible business benefits are ultimately far more illuminating than the debate about the respective merits of each approach.
It’s all about how the approaches can help in streamlining and simplifying business architectures so that organizations can focus on leveraging their data in meaningful ways that deliver tangible business value. Over time, we would expect to see further evolution of the two approaches with data mesh innovations in areas like domain-oriented data ownership coming together with the increasingly mature data fabric architecture. All the time though, the pragmatic focus must remain on what this combination of capabilities delivers to the bottom line. For too many organizations, data infrastructure is still seen as a cost center, but these new paradigms are paving the way for a new understanding of its value, allowing it to be appreciated in a new light as a profit center that contributes its own substantial value to the business.
