In part two of their Energy Transition Talks conversation, Doug Leal and Peter Warren dive deeper into the concept of Data Mesh and its impact on organizational structure. Building upon the first instalment of their discussion, they examine how Data Mesh enables business agility and AI innovation while necessitating a cultural shift, robust data governance and collaboration between IT and the business.
Catch up on part one of the conversation here.
Data Mesh represents a significant cultural shift in how organizations manage and use data. Traditionally, data ownership has resided within IT departments, but Data Mesh advocates for decentralizing this ownership to various lines of business teams.
- Domain-oriented decentralized ownership: Data is no longer solely owned by IT; instead, it allows teams closest to its creation to take ownership and responsibility for its quality and reliability.
- Data as a product: Organizations are encouraged to treat their data sets as products, prioritizing data quality, usability, and timeliness, while focusing on how they can create value from them.
- Self-service data platforms: With multiple domain-oriented data platforms emerging, automation is key, and teams need to ensure these platforms are user-friendly and efficient. The goal is to remove bottlenecks and accelerate data sharing and collaboration.
- Federated computational governance: This model supports governance tailored to specific domains rather than a one-size-fits-all approach, allowing for more relevant oversight.
The transition to decentralized ownership empowers business teams to take control of their data, fostering agility and responsiveness to market needs. However, it also increases their responsibility. Data governance is paramount for Data Mesh. It ensures data quality and security across decentralized domains, creates trust and consistency in data usage and balances autonomy.
“Data quality is still a cornerstone of a Data Mesh platform,” Doug says, explaining that developing this domain-based data architecture requires a robust data quality framework. This involves ensuring data traceability and conducting rigorous quality checks for accuracy, completeness and consistency so organizations can build trust in their data.
Collaboration between technologists and business stakeholders is essential for identifying the most accurate truth as organizations integrate multiple source systems into their Data Lakehouse. This foundation is also critical for future advanced analytics, machine learning, and AI initiatives.
Considering the varying cultural approaches to data management, Peter posits whether data needs to be flawless before initiating a project and whether it is best to invest in equipment or processes first.
Doug recommends adopting an agile mindset and focusing on quick wins: “What is the use case that we can start, deliver the maximum value to the business, and do it quickly?” With the advent of cloud platforms, he says, using Platform as a Service (PaaS) and Software as a Service (SaaS) for lakehouses has become increasingly popular, allowing organizations to quickly establish the necessary infrastructure and focus on a data set that delivers the most value for the selected use case.
The aim is to establish a trustworthy "golden record" dataset, consistent with mesh architecture principles, that enhances and refines even incomplete data. Prioritizing data governance and quality will enable the creation of a data product that serves the entire organization effectively.
Adopting a Data Mesh framework can lead to greater efficiency and responsiveness in handling data, but Doug acknowledges that it requires a fundamental change in organizational culture and processes to be successful. He recommends organizations explore and learn about Data Mesh, citing successful implementations and available public case studies as evidence of its potential benefits for data-driven decision-making and innovation.
Listen to other podcasts in this series to learn more about the energy transition
Read the transcript
Peter Warren:
Hello everyone and welcome back to a second part of a great conversation I'm having with Doug Leal, talking about applying data strategies and data analytics strategies in the energy and utilities marketplace. We've talked a lot about utilities, but these things overlap well into the oil and gas area. We've got examples on that too that we can talk about as well. But Doug, do you want to reintroduce yourself first?
Doug Leal
Yeah, absolutely. Doug Leal, I'm a technical vice president here with CGI. I more than 23 years of experience in the data and analytics field. And I'm one of the leads here of our practice in this area. And I'm very happy to help our clients. We're serving our clients here in the southeast area of the United States.
Peter Warren:
Thanks very much. And in the first part, you talked about the Lake House approach and Data Mesh, and you gave a teaser in that conversation about organizational structure. You said, we'll talk about that further. And that's why we're back here in this part two.
Peter Warren
You brought into a concept about the whole about Data Mesh. you gave probably a bit of a bombshell for a few of our people of handing over the keys of data to lines of business versus keeping them close to the chest in the IT department. Why did you say that? What's your sort of background on that thought?
Doug Leal
Yes, yes, no, absolutely. It is a change. It is a cultural shift that Data Mesh brings to the table. Just a quick recap, the four guiding principles is that the main oriented decentralized ownership of the data. This means that IT doesn't own the data anymore. The data platform goes to the business. And then to start thinking about your data as a product. Think about your AMI data, your AMI readings. How can I create a product out of that data set? And then self -service data platform, data platform that now since we are going to have many different data platforms, domain -based data platforms, how does that align and how is that platform quick, if you will, for us to create. And a federated governance model. We have an enterprise governance model, but this federated governance model allows us to fine the data governance based on the domain based data platform.
It is not for everyone, because this approach, you transfer the ownership and you transfer the data to the business. The business now is on the driving seat. However, there are a lot of responsibility that comes with it. The prize, if you will, is business agility. Since the business now has access to the data, they are in the driving seat and can move quicker. Of course, a well -established data governance process needs to be in place in order for this to work. And also, when we start talking about why organizations are exploring Data Mesh, some organizations, they just failed deploying a centralized data model. Think about big organizations, because Data Mesh is not better suited for small organizations.
But big enterprises, they might have failed deploying a centralized data model or a centralized data management system. And that's where they are looking for this federated approach. And also the business accountability of the data. Many business, they want access to the data. They say like, hey, I have the use case. I have the technical skill set, which is very critical for Data Mesh to be successful. You need to have the technical skill set as part of the business team, but it is critical for the business to have that accountability over the data.
Peter Warren:
Yeah, again, sort of going back to some history here is just looking at it. So again, in our Voice of Our Clients survey this year, which involves interviewing our clients around the world and other people as well, potential clients and new customers. We asked them two questions. One was, are you investing in this type of area? That's, and those that said yes, we asked them a follow -up question. Are you getting the ROI? Your intended or inspected ROI on that. And there was an interesting correlation between that, that the ones that said, yes, I'm getting the ROI back I expected versus those saying I'm investing it, but I'm not seeing the return was actually a cultural one. One where they actually culturally were more nimble, more agile, and were able to manage things better. It also got back into the fact that you said they're putting in better tools and better, it's not a technical tool, but you talked about before more of a structural tool to fix that data. Do you want to expand upon that?
Doug Leal
Yes, absolutely, because data quality is still a cornerstone of a Data Mesh platform. As we build this Data Mesh approach or this domain -based data platform, we need to make sure that we have a data quality, a strong data quality framework in place to as we bring the data in right from the system of records, we are doing data quality checks, making sure the data can be trusted. Because you can build the best data platform architecture. If the data cannot be trusted, nobody's going to use it. As the old saying goes, garbage in, garbage out. And with that approach, we noticed that most successful implementations of Data Mesh, they leverage a decentralized operating. And here's what I mean by that is this operating model is where we have the technical members of the team as part of the business. Of course, we still have IT. IT is still enabling Center of Excellence and managing the platform. But we have data scientists now as part of the IT team. And we have data engineers as part of the business team. In this more decentralized or functional operating model, the business has been able to be more successful addressing data quality, right, and implementing a framework for data integrity.
Peter Warren
So there's bound to be conflicts with data. One system says, I live here, and other system says, I live there. There could be two people with the same name. Could be two pieces of equipment show up two different places. There's a lot of things that can be confusing between systems. Everybody believes their system is the system of record.
Doug Leal
Absolutely, yes.
Peter Warren
How do you resolve those things? How do you structurally put those things to bed?
Doug Leal
Yes, that's a great point. And the answer to that is a strong data governance framework. And when I mentioned data governance, it's not only on the data quality side, but also think about data traceability, data lineage. Because this will pay off in the long end, once you building advanced analytics, machine learning, AI.
Now you already have the foundation in place to address your model explainability requirement. So having a process, a strong framework in place in your lake house that you can enable data traceability and especially data quality. And when I say data quality, it's data quality checks to ensure accuracy, completeness, and consistency of the data. And all of that in collaboration with the business, right?
We need to rely, as technologists, we rely on the business to help us and collaborate with us. It's like, what address is the most accurate or version of the truth? And as we're integrating dozens and dozens of source systems into the lake house, we can apply those rules to make sure that data can be trusted.
Peter Warren
A bit of a loaded question here, because you hear this a lot and culturally different parts of the world are diving into this and other parts are being reserved. Does the data need to be perfect before I get started or how is this something that I move forward and do I buy the equipment first or do I put the processes in first? What's your recommendation?
Doug Leal
That's a great question. Another great question. And I'm a big fan of agility, I believe, and the low -hanging fruit. What is the use case that we can start, deliver the maximum value to the business, and do it quickly? And with the cloud platforms, we don't need to commit to buying an expensive database appliance.
We can use software as a service, which is very popular nowadays for lake houses. We can start fairly quickly by standing up the infrastructure on your preferred cloud provider. And from there, work on the value, what data set that brings the most value for that use case. And as I mentioned on the previous episode, the lake house has different layers.
So even if the data is incomplete, we can bring this data in to what we call a raw layer and address the completeness, the accuracy, and the consistency of that data in the lake house as we move the data to higher layers of the lake house and until ultimately, right, bring this data to a golden record, right, to this record that can be trusted. And going back to the mesh architecture, if you will, or methodology, that's when we get to a data product. We bring the data in, which might not be complete, but we go through the process of transforming, addressing data quality, data governance on top of it, so we can build that data product, which can be leveraged by the entire organization.
Peter Warren:
That's great, Doug. While we're at time here, I'll give you a chance to give a final thought. If you had to summarize all your thoughts into a couple of different points, it's a tough question, I know. What would you like to leave the audience?
Doug Leal
Well, I would like to touch back on Data Mesh. I know sometimes it sounds like a buzzword, but there is value in it, but it is not for everyone, right? So I would learn about it, read about it, see what it's all about, because there's definitely, we have been part of successful Data Mesh implementations for our clients, right? There are some public case studies out there that shows that can be successful.
However, it is hard. It is not for everyone. The operating model of your organization needs to align with this federated approach that Data Mesh proposes. So, I would like to close with that. Again, thanks for having me. It's a very exciting time to be working with data and I'm looking forward to continue helping our clients in this area.
Peter Warren
Appreciate it, Doug. Thanks very much for your time. Bye, bye. Bye, everyone.
