Mastering Chaos - A Netflix Guide to Microservices

[@sundeepreddythirumuru5266]

One of the best talks I have seen on building systems of high scale, high availability, failure resilient and agility in undergoing change.

[@StingSting844]

Can't express how high quality this talk is. Wonder if he was a teacher before!

[@jordantaylor8549]

Yah I was wondering the same thing . Very top tier presentation and speaker

[@JamesJansson]

When you are the lead developer for a large org, it is your job to be an educator at a high level.

[@ThunderAppeal]

Youre full of shit. You have no idea of any of this.

[@yahorsinkevich4451]

I would say this is a perfect talk that describes how to writer microlyth and how NOT to design microservices based arch

[@StingSting844]

@@ThunderAppeal Yeah thats why I have come here to learn!

[@OliFubar]

I am sad I found this 3 years late, I've been developing microservices for the past 3+ years but I still learnt a lot from this. A huge belated thanks to you Josh!

[@sairampareek]

And I found this 3 years later you commented

[@ashimov1970]

1:51 "how amazing the human body is and how something as simple as an act of breathing or interacting with the world is actually a pretty miraculous thing. And it’s actually an act of bravery to a certain extent. There are so many forces in the world, so many allergens and bacterial infections and various things that can cause problems for us." - watching this on Feb 12, 2021 makes me admire this guy and the company he dedicated part of his miraculous life.

[@arthurkeech]

One of the best technical talks I've ever seen.

[@sandy.aggarwal]

Watched again 3 years later and it's still so relevant.

[@Inquisidor9408]

it doesn't get old

[@souvikghosh8068]

I come back to this video once in a while to remind myself how cool the software engineering can be.

[@UberOcelot]

As someone who doesn't deal with this side of architecture, this is a fantastic talk. Micro-services are just a division scheme, not really an architecture. To get things really running smoothly this provides some great insight to where the real work starts.

[@Patrickdaawsome]

Note to self: Write down summary starting from the 51:25. P.S. I watched the whole thing and absolutely loved it.

[@andrewferguson6901]

in case you didn't write it down where you could find it

[@jonopens]

Such a fantastic metaphor to communicate the intent of microservices. As a younger programmer, that really resonated with me.

[@infoq]

If you are interested in microservices content you can check our collection on InfoQ. www.infoq.com/microservices

[@prakhargupta629]

Few years into corporate job and I had forgotten how many things are there in Computer Science. This talk brought me back to college.

[@ErnstFluttert]

Great talk, which is still relevant in 2020! Especially the practical examples, and the solutions they applied. Loved the analogy with the human body. Thnx Josh!

[@martinmogusu]

This is a really informative talk on microservices, with real challenges and real solutions. The human body analogies make a lot of sense, there's a lot we can learn from nature...

[@killoverflow]

One of the best microservices talk I have heared.

[@tidal4774]

One of, if not the most comprehensive and analogous talks on the subject with relevant, real world stories from someone who has lived it. Thank you.

[@OrdinaryFemmale]

this beast is *ONLY* working for netflix for the last +20 years, that's pretty amazing! thank you for this video, great presentation!

[@drxyd]

It's amazing how quickly the difficulty of engineering systems becomes with scale. My first instinct seeing the architecture was "too complex" but I've never worked on anything so large, would be awesome to see first hand.

[@bdpyne]

Really fantastic talk. He used analogies well, making abstract concepts more concrete. My most important takeaway was the evolving nature of architectures. Architectures are challenged both by external factors (customers) and internal factors (operations, customer support, etc.).

[@markmd9]

I just realized that on the globe we have more than 2 millions people that have interest in microservice architecture

[@jidnyasadhavale6664]

So Informative. Tech talks are the best way to get insight into how huge applications are built, complexities involved, challenges and solutions.

[@a0flj0]

Actually, no. What you get, mostly, from tech talks, is various tech people advertising their pet technology, in a way that appeals to other tech people. If you really want to really get into big systems, find work on one, and do thorough research - tech talks are a good start, but often leave important things out, and only show you the pictures of the car's body, without teaching you to work on the engine. To learn about cars, you go to grease monkey school, not to car shows. The reason for our industry being insanely hype driven is probably at least partially that people expect to actually learn something from tech talks, instead of considering them simple pointers to start research from.

[@gashinamu]

One of the best tech talks I've ever watched.

[@infoq]

Happy you like it. You can find more on www.infoq.com/

[@CodewithRSV]

This should be mandatory viewing for anyone working in a microservice architecture.

[@jegtugado3743]

This talk is amazing. I'm far from being a system architect but this gives me a bigger perspective on how amazing systems like Netflix is built.

[@cthackers]

"The structure of any system designed by an organization is isomorphic to the structure of the organization." Melving E.Conway

[@aceintheblackhole]

one of the best talks i've seen at any conference on any subject. i'm actually using it as a real-world resource for my system design interview preparation because it's very informative in many ways (especially for a beginner in distributed systems like me :) ). thank you for making this available!

[@LesterFD]

copying and learning from nature is always something beautiful. natural sciences like physics, chemistry do it since centuries. It's a good time for computer science to learn from methods, that are applied for many thousands of years.

[@sirnawaz]

Brilliant talk! ... and the references to the various biological systems of the human body, throughout the talk, makes it even more interesting. It is amazing to see how various aspects of microservice architecture find similarities with other things created by the nature itself. On a microscopic level, we can draw analogies to the multiple-core, memory fences and cache-coherence as well.

[@clarkpeng8106]

what I've learnt important from this video is keeping perfecting your tools & architecture.

[@sathvikvutukuri9179]

The way you explained microservices referring human body is mind blowing. Yes nature is the biggest Wikipedia or GitHub of everything. How much we learn or analyse or extract knowledge decides human evolution.

[@thaituandat31]

I understand that Netflix is not only "NETFLIX" now. Thanks for your team, too. Amazing topic.

[@usun_current5786]

This is the most useful presentation I've ever seen regarding large scale platforms. Shared in my org, hopefully we'll lift a lot from here for future extensions.

[@Innovate22]

The Alton Brown of software engineering nailed this presentation 👏👏👏

[@cardboard3161]

Speaker: "Breathing is a miraculous act of bravery." 2020: "Challenge accepted."

[@Yusuf-ok5rk]

he is breathing in micro doses so that he can serve a continuous presentation

[@p3k1n0]

I could watch an entire season of this stuff. Way better than the standard Netflix products.

[@apurvsawant5703]

This is one of the best videos to get insights into micro services architecture.

[@bobross3305]

Thats crazy i never even thought about all this stuff before, because i use pirate bay.

[@antonvedel3028]

His analogies to the human body make great sense. Awesome video. Thank you.

[@somescams]

@Newtube Exactly.

[@madd5]

They do. But a regular monolithic application can also be analogous to a living organism.

[@ybenihime8757]

Awesome stuff, after watching this in 2020, i got myself thinking what changed in the architecture to this actual year.

[@the.abhisheksinha]

nicely explained .. I wish Netflix gets all such videos on Netflix and show us how do they architect and design their mammoth system

[@thewebmaster1]

When adding a single table becomes a cross-functional problem, then the architecture suite of microservices is messily constructed, making backend administration of Data, correcting, adding or removing data, a complicated task instead of a simple 5 line operation. Middle Tier and Platform Services will one day cause the entropy of the Matrix, for a matrix it is. It is an Architecture that can one day fall like dominoes in many possible zero day scenarios

[@lexdemonica]

Such valuable insight from an articulate and knowledgeable insider. TYVM

[@kibiz0r]

AKA: Obfuscating Chaos - A Netflix Guide to Creating Dysfunctional Organizations Basically, a dev noticed: Hey, these two components could benefit from having isolated processes. And a manager said: Wait, could this also be used to stop Team A from complaining about consuming Team B's code? Thoughtful dev replied: Well, it depends on the specific components... Some components are fundamentally coupled, so they can't survive a partition anyway, and introducing one for the sake of team boundaries will just create more technical problems without solving the underlying social problems. Manager blinked twice, and after he rebooted he said: But they'll shut up about each others' code? Dev: I mean, technically. They'll just complain about API contracts instead -- Manager: Perfect. Dev: -- which is arguably worse, because as soon as you have a multistep process or something involving callbacks or dependency injection, you have a huge mess, and that's even if you can sidestep the CAP theorem for all of your use cases, which is a really big assump-- Manager: Yeah, yeah, whatever. So how soon can we do this?

[@chrisalister2297]

Built my system on MS ASP back in 2002. I was already using something more or less like micro-services and using ASPCache(like EVCache). Competitors would have issues/shut downs while mine systems kept trucking along with very little down time. Think of worker ants in colonies. All working towards a common goal.

[@joechen2800]

Using organization as examples are really self-explanatory. Brilliant talk!

[@andrewmadge]

Great video! thank you for putting this on youtube

[@igrai]

very good and a very open talk, no generalities - really an insider's look into these challenges - awesome!

[@RonMartinJr]

I would love to just sit in a room with this guy and listen. Such a wealth of knowledge and experience.

[@prashanth0458]

I read the news of netflix introducing own bugs into their own system to make it more efficient long back, today I am watching how it is done. Good video with latest technological stuff like redundancy is good, bugs infection, support for multiple languages, region server failure routing etc..

[@deepsue]

Thanks, this analogy, and learning is still happening and needs to be implemented in multiple applications/organization

[@chungyuwaiki]

Great talk and hope i will understand much more years later.

[@OwaisAhmedMuhammad]

Simple and valueable. A very nice and practical talk on Microservices

[@himanipku22]

Thanks for reminding me I don't have to think about breathing.

[@Joe99]

On the topic of client libraries. A lot of recent talk argues that we should keep things DRY within a microservice bounded context, but not between bounded contexts. This introduces coupling and client libraries are one example. In the .NET world these shared client libraries can also lead to some nasty dependency versioning hell conflicts. I assume this issue is the same in the Java world, but I have no first hand experience. Oh yep, he mentions it at 21:07.

[@Walruz1000]

I read the same thing and am a bit confused by that as it seems to introduce a degree of coupling which in my limited studying seems to be something to avoid.

[@usun_current5786]

Yep, similar issues in Java world

[@jasonamaroreinert4499]

Great work. I know most of all of this content (whoopdie do I know), have been building microservices (via a standard reverse proxy/load balanced REST API-first architecture) since 2007, and yet this is still an amazing watch and important to keep instilling. That's because knowledge isn't a one-time thing, or a time-oriented thing at all but rather a means to solidifying a way. I love that this content is out there and presented this well because these stories are still happening out there even 15-20 years later!

[@Larry21924]

This is absolute perfection. I recently read a similar book, and it was absolute perfection. "Mastering AWS: A Software Engineers Guide" by Nathan Vale

[@Mike.e]

Excellent talk. Envious of your delivery! Thank you!

[@tobiasfuchs7016]

2:33 This visualization so beautiful and mesmerizing, and you immediately get the human body analogy. The whole presentation sure was hard work, and well worth it.

[@guseynismayylov1945]

it's not accurate though, it shows that microsevices are one directional, which is not true.

[@yao855]

Kubernetes was announced in July 2015, so I wonder why he said there's no good node orchestration tool out of the box at Netflix. Maybe they take extreme precautions when adopting new technology. Any insiders can confirm how they do it in 2020, did they build out in-house infra for that?

[@a0flj0]

I'mmalso curious what they are using. But, as a hands-on user of k8s, I think he's right about the no good orchestration tool. K8s is widespread and provides an abstraction layer for the actual cloud provider you are using, preventing lock-in. But it isn't actually that good. It has higher latencies than would be ideal, by design, caused by its model of constant sync of resource definitions to actual state. It keeps failing in unexplainable ways, even without unleashing a simian army upon it - luckily, rarely catastrophically. It provides low level support but no high level functionality for devops patterns that become increasingly expected/state of the art, such as blue-green deployments, canary deployments, continuous integration and continuous delivery. Its autoscaling capabilities are similarly limited, and autoscaling is also slow. Ops people are unecessarily exposed to many internal details. It's not outright bad, it's just quite far away from ideal, or even good. Hashicorp's Nomad is a much better solution, technically - it does less, but what if does it does better, and it lets you integrate things that solve additional problems easier. And it is more flexible in what it allows you to do - by far. These two aspects allow you to build infrastructure with a lot less cluttering, which is easier to analyze and reason abou, where coupling between different systems is lower. Unfortunately it's not as wide spread, and not as frequently available as a service. And it doesn't have a similarly strong marketing. But marketing alone doesn't make k8s good.

[@kim-uk1wv]

For shared client libraries. What about requiring each service to make their own wrapper classes/library that only exposes the parts they need. Then, to make it safer against changes, they also create their own unit-tests for their wrapper classes? Or if they use almost the entire client library they could just create their own set of unit-tests for the original library and skip the wrapper. When a shared client library is changed its test runner runs all the different wrapper library unit-tests. Then if anything brakes they can just contact the team responsible for the service owning the broken tests and work out a solution together.

[@a0flj0]

How do you solve API versioning and transitive dependency conflicts in the same library? He explained pretty well what the problems with shared libraries is.

[@Nampjg]

This man is genius... He explains things so well... It was enlightening to know that many of netflix's architecture optimizations are taken from Biology

[@andreelyusef3235]

We need more of these kind of lectures. Spot on sir please do contribute more.

[@dneary]

Interesting aroun 12:45 - updating the cache is, in his presentation, the responsibility of the service layer - I would have expected that to happen in the client library (AKA: check cache. If (miss), request result from service, store it in cache, and return it to user).

[@yanikjayaram]

18:25 - hey guys does anyone know what software can produce this animated representation of a backend system? Would love to represent my own system this way to onboard other engineers easily.

[@CoolHandMikeYT]

This is a really great talk to understand problems and solutions with microservice oriented archtecture at a large scale.

[@not-high-on-life]

Fun thing is what was applicable to netflix(read usage of microservices) may not be applicable to your company/startup. A lot of software developers don't realise when something is applicable and start to put this shit everywhere they can reach... Exactly this is happening with crypto and microservices right now - hype oriented development, they hear buzz they look how cool it is and instantly start to put everywhere...

[@anon_y_mousse]

When he mentions virtualization for dynamic resource allocation, is he talking about VM style virtualization, or more like the Erlang style of dynamically dispatching processes? I would assume the latter, but want to know how others interpreted it.

[@VivekKartha]

Uncle Bob always talks with so much clarity. He has a deep understanding of all the topics he presents.

[@ahmadbaiazid976]

probably one of the best guides on youtube for microservice arch

[@mattcargile]

Man! I really feel sad watching this like I missed the technology boat to be a part of something incredible! Like my life will never mean as much as this talk.

[@SartajHundal]

Turning Occam's Razor into a lecture. Well done.

[@timriley1501]

How can anyone even give a thumbs down to this? Very interesting and informative video.

[@midnightsyntax3333]

Excellent presentation! The slides are super clear and specific. 5/5 kernels

[@ronfisher4751]

Very good stuff Josh you are a smart man and a great presenter of complex information

[@ChidambaramVelayudham]

Absolutely Awesome. Love the way how Josh presented. Appreciate you for sharing the knowledge.

[@ankitsorathiya2304]

Thanks Josh, It was an insightful talk, Thanks for sharing how netflix evolved over period of time.

[@DodaGarcia]

Surprised by the COVID cameo at 3:00 - always feels weird to see celebrities before fame

[@ralphe9668]

wow thank you for this clear explanation of complex arcatechture. it confirms my theory of possible manipulation to the market demand rather convert manipulation for the highest bidder. just my opinion. what is your opinion?

[@dorcohen3522]

A disclaimer: if you don't have to scale different parts of your application independently, don't bother to go microservice. It won't solve any of your issues. Splitting an inherently coupled monolithic application will turn it into a distributed one, which is worse by any metric. Domains should be loosely coupled with or without microservice architecture. If your monolithic application is modular enough, there should be no problem to deploy it separately whenever needed.

[@srolesen]

I liked the 2017 version of netflix so much better, but thanks for this, fantastic presentation.

[@seanastro5549]

Just awesome... but wanna see the 2021 version now

[@bhanuxhrma]

What are the client libraries? How does one micro service call another one. If it don't call another one using rest endpoint then how does it call

[@lepidoptera9337]

What do you do when your business plan is, essentially, the sale of CPU cycles? You offer products that use as many of them as possible to achieve as little as minimally required. Enter microservices. Genius.

[@KentMarete]

Quality talk Josh. Question Josh Evans, what do you do when an entire region is compromised like succumbing into hacking activities? Do you kill the region fail over into the others to fix the vulnerability?

[@asdfasdfasdf12]

hey guys, whats the alternative to the vizceral tool they were using for visualization?

[@parisnakitakejser]

Very nice video about microservices, how you can split up the traffic between services and scale databases behind the microservices its very hard to do, but the code split its "more easy" but thanks so much for this nice video, :)

[@raghavendrakrishnamurthy4041]

Awesome presentation!! Top class!

[@andreeadianaratiu5091]

Great presentation and nice analogies.

[@tmmyhmmie7761]

does anyone know what tool(s) can do visualizations like in 10:45? seems much better than scrolling through log files

[@wartem]

good stuff

[@frustratedmajority851]

The Theta Network is going to revolutionize how CDNs deliver content. It's a decentralized CDN network consisting of millions of user's smart devices acting as edge nodes that cache, encode, and stream video to those who need it... taking the work load off of the centralized CDNs (Netflix, KZbin, etc). This not only saves these centralized CDNs money in operational costs, but also improves the speed and quality of the video content by eliminating the "last mile problem".

[@markusklyver6277]

@18:17 I think the math is wrong, the different services are not independent of each other and hence you can't calculate the probability of a failure with a simple multiplication.

[@OneMinuteNotes]

Its one of the best story regarding microservices implementation. Cool

[@LuisEDITS_KLK]

really good talk still in 2024 if you're new to microservices

[@bpate1138]

Where can I find a dashboard for taking in traffic? like shown in video

[@varunvora]

I'm wondering where they ended up with the debate on client libraries. I've certainly come across situations where I feel they are essential and some cases they are a burden.

[@AsimMittal]

One of the best tech talks period

[@iwisdominheritance]

levelsetting, so that we have same vocabulary. challenges and solutions relationship between organisation and architecture

[@Deepak_Tao]

Thanks Josh...very nice presentation!

[@cornel4931]

What do you use to visualise networks like that?

[@cinemarob1]

2:04: "There are so many forces in the world, so many...bacterial infections and various things that can cause problems for us" - Ahead of its time

[@DodaGarcia]

And then bam, surprised COVID image at 3:00 lmao

[@cinemarob1]

@@DodaGarcia Loom'nati!

[@soumyachatterjee4135]

16:29 The vaccine doses :P This went to spooky level