What Is Data Mesh, and How Does it Work? ft. Zhamak Dehghani

[@michaeldemarco82]

This is fine but truly what she is talking about is one of the benefits of Microservices anyways so nothing new from my perspective just evangelizing it is what is happening here. There are two problems with Micro services depending on where they are located. Data across the wire is latent so Microservice does not solve that. This is why compute and storage is great when they are colocated (but there are drawbacks to even this). Data latency is one of the biggest problems in performance so how is that solved? Also, another problem with Microservices in a purist sense is that when your doing reporting and you need data aggregated Microservices is not the solution which is why there are hybrid solutions that have Microservices but then data is aggregated in the repository and indexed for quick response by a push to the data mart which updates in the background. Try to do a query across Microservices data. The core problem with data will always be that the goals of singleness of purpose and distribution of that data will always be in conflict with speed and performance which is why you need a hybrid solution with data pulls from Microservices to fulfill consumer needs but for reporting and analytics you need data aggregated and delivered fast which is where the updates to the data mart come into play. you can lessen the forking effects of distribution and performance with query engines like hyperscale or big query but your going to pay dearly for it

[@brookster7772]

My first thought is the data mash is just another layer on top of everything else that has no API and dozens of client applications pulling data directly from the tables in the data mesh….? Almost takes me back to the 90s where we just had a massive oracle database and every department had their own clients getting SQL connections and running SQL queries… later of course we close the database behind web services and now we seem to be going back to the open massive data mesh? Still a little unclear to me, but it’s some thing I need to understand and learn and looking forward to it.

[@mailmahee]

not all data-lakes are necessarily orthogonal - s3 bucket access policies and IAM can accommodate most of the points Zhamak is making but would involve a platform paradigm shift as well as a cultural shift in {k}ontainerized ops practises.

[@brookster7772]

I would be very interested in learning more about this new data, mesh architectural layer, and the connection to the AI engines such as Palantir for example… Palantir has an abstraction called the ontology… how does all of this shit together? I am an architect, tasked with implementing these new technologies, and I am very interested in learning more if somebody could make a recommendation

[@mainajnabee]

Thank you for asking right questions and picking her brain on this topic. She's a treasure trove! 👏👏

[@sweeper240]

Complimentary to any warehouse lake, lake house, or mesh... or whatever new concept of data storage governance and security model is thought up in the future Our AI/ML effortlessly harmonizes and contextualizes all data across all silos even third-party or unstructured content like handwriting, images, etc If you want to achieve mesh at any scale, you'll want to talk to me.

[@reactiveland3111]

It's just another microservice, another node in the streaming topology ... These "new" microservices own/process/publish analytical data. And bringing all the data HOME (domain team) from AI team

[@michaelburton8024]

Tim I have a free Confluent acct so how might I use the code for extras? Thank you

[@Confluent]

Hi Michael, this document may be helpful in answering your question: docs.confluent.io/cloud/current/billing/overview.html#promo-codes

[@michaelburton8024]

@@Confluent Thank you so much for the prompt help

[@dlinanhu]

I dont get the idea of Data as a Product. Can you give some examples please?

[@Confluent]

Hi David! This resource on data as a product from the Data Mesh 101 course on Confluent Developer might be useful: cnfl.io/data-mesh-101-module-3. If you have additional questions, feel free to post them at forum.confluent.io as well. Hope this helps!

[@dlinanhu]

@@Confluent thxs!

[@mehdismaeili3743]

Excellent.

[@ankitlakum1]

🙏

[@manishchandrajiit]

Thanks. This video gave me really good amount of idea on what it is, why this concept came about and challenges ahead.