What is a Data Mesh | Data Mesh explained

This video explains the concept of data mesh and its characteristics.
What is a Data Lake: • What are Data Lakes | ...
What is a Lakehouse: • What is a Data Lakehou...

[@chanakyamishra4795]

You are covering the topic comprehensively within 5-10 mins .... That's awesome.....Thank you so much.....Appreciate it

[@BigDataThoughts]

Thanks Chanakya

[@ambar752]

To summarize, what the Datamarts are for a DataWarehouse, same are the DataMesh for a DataLake

[@kannanlg4979]

But once these data domains and it's corresponding ownership is created, how will it serve data scientists who need collective raw data to work with.. I see it's similar to have different schemas per domain in a data warehouse.. I kind of see a lot of overlap.. probably the only difference is the ETL can be handled by the domain team or can still be managed by a centralised team based on needs.. also how can it complement with some thing like a data lakehouse?

[@AmitGupta-ok2wq]

Shreya what Cloud Data Mesh Platforms available today for enterprise grade today to manage TB's of data and scalable upto petabytes

[@muralichiyan]

Data mesh and snowflake same..? Data mesh and microsoft fabric same?

[@pushkarchavan]

Are we going to the old way again having separate data (lake or function tables) connected by primary or foriegn keys relationships?

[@artemgri9188]

this is DDD concept which says to have own copy of data in each domains suitable for this domain, since 2012 by Evans. It's not new concept "data mesh".

[@ramdasnarayanan3345]

Aren't data domains similar to data marts, where each of them is owned by a set of people?

[@srirambalasubramanian1627]

I suppose data marts are built on top of a CDS which is a common database, extracting only the required information specific to the domain. Wheares, here in mesh, there is no such CDS from which individual data gets pulled, rather the data is sourced directly to the domain (Only the required data specific to that domain)

[@chandrashekharkotekar8453]

Hi Shreya, thanks for the video. How do we join data from multiple data domains? Usual ETL process or something else?

[@BigDataThoughts]

In data mesh we can get data from different domains and then we can join and utilizes as per need

[@Flavoursandfrequencies]

hey chandra , we creat GUID to join different domain data .if a consumer would want data from different domain for specific then based on guid the data is extractes from different domains

[@Watson22j]

very nicely explained, Thank you! I have one doubt though, since we are storing data domains separately managed by different data owners/teams, do we still have one pipeline? or every domain works on their own pipeline?

[@sandiproy1279]

Nicely expliained Shreya!

[@BigDataThoughts]

Thanks sandip

[@SQLwithManoj]

Nice introductory video about #DataMesh...You explained about how the data is managed here, but can you provide some idea on how the data will be served or retrieved from diff domains?

[@luisgutierrez9248]

thank you Ms.

[@HariharasudanP]

Hi Shreya, what about having different schema and storing data as different domain(sql DB) compared to having data domains that you are talking about ? Any example pls to correlate and understand more ? (Thanks again for video on Monday, I thought no video for today. But you haven’t missed it. Thanks once again)

[@BigDataThoughts]

We can have data in different schema or different storage. But the essence is that we are storing data domains seperately managed by different data owners/teams rather than centrally with one ELT pipeline. The concept is to not have a central storage that may not be flexible in terms of adding new data source.

[@KunalKumar-yr8bm]

I am a recruiter I need a data mesh candidate for the location at connecticut if you have please let me know

[@jamesbenedict6480]

I am more confused - in the past, various "islands" of data was causing all kinds of headache for the business to consolidate reports....with the consolidated data domains, things were much easier to rely on as they were catagorized as "single version of the truth"... well....now comes Data Mesh which is encouraging (so to speak) to seggregate the single verson of the truth and distribute is across the enterprise?

[@srirambalasubramanian1627]

Thanks Shreya for the knowledge shared. Truly helpful. Had one question though. Is that only with the help of global governance team, one domain data can be accessed/understood by a different domain in case required. Or they have to create/duplicate a separate metadata on their own domain despite the presence of the same in a different domain instead of reusing that

[@BigDataThoughts]

Thanks sriram. Data duplication may occur as there may be common datasets across domains. Metadata can be managed centrally. Here we are moving away from centralized storage to domain driven disparate data storages.

[@srirambalasubramanian1627]

@@BigDataThoughts , Thanks for the quick response. If metadata is central, will that Entity hold all the associated attribute from each of the domains? Or only the common ones, leaving the domain specific attributes to be defined individually

[@sahil0094]

Can you pls make a video of implementation of data mesh, as per comments I see people still have a lot of doubts like how will data inter domain will be accessed, will there be duplication etc

[@autodiddy]

This was crazy helpful

[@BigDataThoughts]

Thanks Rahkeem

[@thesadanand6599]

Better if you could help with an example

[@maryammoslemi]

Good comparison!

[@BigDataThoughts]

Thanks Maryam

[@amitgtalks]

found it helpful. You may go slower though. I had to stop and rewind few times.

[@mithunnambiar1433]

This technology is still at it's evolving stage right? Isn't this the same that Zhamak Dehgani advocates upon?

[@BigDataThoughts]

It's more of a concept than technology

[@user-hr5bu9nk9k]

You are awesome

[@BigDataThoughts]

Thanks

[@ndz7372]

Thanks