Data - The Land DevOps Forgot • Michael Nygard • YOW! 2023

This presentation was recorded at YOW! Perth 2023. #GOTOcon #YOW
yowcon.com
Michael Nygard - General Manager of Data at Nubank & Author of "Release It!" ‪@mtnygard‬
RESOURCES
/ mtnygard
/ mtnygard
www.michaelnygard.com
ABSTRACT
We've transformed our operational systems to microservices with stream-aligned teams that own what they build. We know that working in short cycles with fast feedback produces better value sooner, safer, and happier. But there's a dark underbelly. Those microservice databases get pulled out into analytics processing that runs in monolithic batches that run for hours on hundreds of cores, then break with an error. Feedback takes days, and the centralized data team can't keep up with the pace of the business or data growth. There's another way, gaining traction. Data mesh promises to re-decentralize data processing along with ownership. It applies product thinking to internal data products.
We will talk about moving to data mesh: the promise and the pitfalls, what you can buy and what you'll need to build, and how to get internal buy-in for the transformation. [...]
TIMECODES
00:00 Intro
02:11 Operational vs analytic
06:35 The opposite of DevOps
08:36 Even more challenges
09:36 Hope on the horizon
11:07 Is that actually a new paradigm?
11:27 Data mesh
13:22 Decentralize ownership
16:36 Good fences make good neighbors
24:20 Simple idea...hard to implement
34:31 Missing parts
38:10 How to get there?
45:03 Summary
46:10 Outro
Download slides and read the full abstract here:
yowcon.com/perth-2023/session...
RECOMMENDED BOOKS
Michael Nygard • Release It! 2nd Edition • amzn.to/3WJeKV8
Michael Nygard • Release It! 1st Edition • amzn.to/3XCkiRf
Zhamak Dehghani • Data Mesh • amzn.to/3tTCwAC
Eberhard Wolff & Hanna Prinz • Service Mesh • leanpub.com/service-mesh-primer
Piethein Strengholt • Data Management at Scale • amzn.to/3tya08H
Martin Kleppmann • Designing Data-Intensive Applications • amzn.to/3mk2Roj
Sandeep Uttamchandani • The Self-Service Data Roadmap • amzn.to/3wAw5W2
/ gotocon
/ goto-
/ gotoconferences
#DataMesh #Data #Microservices #SoftwareArchitecture #DevOps #MicroservicesOfData #Decentralize #SLO #MichaelNygard #YOWcon
Looking for a unique learning experience?
Attend the next GOTO conference near you! Get your ticket at gotopia.tech
Sign up for updates and specials at gotopia.tech/newsletter
SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily.
kzbin.info...

[@user-bk4rv6qx2z]

Awesome talk. Thank you.

[@Kantares71830]

Great talk!

[@stephendgreen1502]

How do you do analytics on eventually consistent data?

[@lkjhoiuy97yjhgghfyrthgvjhguty]

Eventually!

[@NtExtazy]

Depends what kind of analytics. Often, you don't need to do analytics on live data. Someone runs a query/analytics, interprets the data, makes conclusions and proposes changes/improvements. Usually it is run on past data at which point eventual consistency is not a big deal. Eventual consistency might become an issue in near real-time systems where you have apps that react on data real-time. For those use-cases you could use frameworks like Flink that has powerful state management and window functions, allowing you to do near-real time stream processing.