Mastering Chaos - A Netflix Guide to Microservices

[@StingSting844]

Can't express how high quality this talk is. Wonder if he was a teacher before!

[@jordantaylor8549]

Yah I was wondering the same thing . Very top tier presentation and speaker

[@JamesJansson]

When you are the lead developer for a large org, it is your job to be an educator at a high level.

[@ThunderAppeal]

Youre full of shit. You have no idea of any of this.

[@yahorsinkevich4451]

I would say this is a perfect talk that describes how to writer microlyth and how NOT to design microservices based arch

[@StingSting844]

@@ThunderAppeal Yeah thats why I have come here to learn!

[@sundeepreddythirumuru5266]

One of the best talks I have seen on building systems of high scale, high availability, failure resilient and agility in undergoing change.

[@arthurkeech]

One of the best technical talks I've ever seen.

[@OliFubar]

I am sad I found this 3 years late, I've been developing microservices for the past 3+ years but I still learnt a lot from this. A huge belated thanks to you Josh!

[@sairampareek]

And I found this 3 years later you commented

[@ashimov1970]

1:51 "how amazing the human body is and how something as simple as an act of breathing or interacting with the world is actually a pretty miraculous thing. And it’s actually an act of bravery to a certain extent. There are so many forces in the world, so many allergens and bacterial infections and various things that can cause problems for us." - watching this on Feb 12, 2021 makes me admire this guy and the company he dedicated part of his miraculous life.

[@killoverflow]

One of the best microservices talk I have heared.

[@ErnstFluttert]

Great talk, which is still relevant in 2020! Especially the practical examples, and the solutions they applied. Loved the analogy with the human body. Thnx Josh!

[@Patrickdaawsome]

Note to self: Write down summary starting from the 51:25. P.S. I watched the whole thing and absolutely loved it.

[@andrewferguson6901]

in case you didn't write it down where you could find it

[@sandy.aggarwal]

Watched again 3 years later and it's still so relevant.

[@Inquisidor9408]

it doesn't get old

[@gashinamu]

One of the best tech talks I've ever watched.

[@infoq]

Happy you like it. You can find more on www.infoq.com/

[@martinmogusu]

This is a really informative talk on microservices, with real challenges and real solutions. The human body analogies make a lot of sense, there's a lot we can learn from nature...

[@jonopens]

Such a fantastic metaphor to communicate the intent of microservices. As a younger programmer, that really resonated with me.

[@infoq]

If you are interested in microservices content you can check our collection on InfoQ. www.infoq.com/microservices

[@tidal4774]

One of, if not the most comprehensive and analogous talks on the subject with relevant, real world stories from someone who has lived it. Thank you.

[@jidnyasadhavale6664]

So Informative. Tech talks are the best way to get insight into how huge applications are built, complexities involved, challenges and solutions.

[@a0flj0]

Actually, no. What you get, mostly, from tech talks, is various tech people advertising their pet technology, in a way that appeals to other tech people. If you really want to really get into big systems, find work on one, and do thorough research - tech talks are a good start, but often leave important things out, and only show you the pictures of the car's body, without teaching you to work on the engine. To learn about cars, you go to grease monkey school, not to car shows. The reason for our industry being insanely hype driven is probably at least partially that people expect to actually learn something from tech talks, instead of considering them simple pointers to start research from.

[@UberOcelot]

As someone who doesn't deal with this side of architecture, this is a fantastic talk. Micro-services are just a division scheme, not really an architecture. To get things really running smoothly this provides some great insight to where the real work starts.

[@OrdinaryFemmale]

this beast is *ONLY* working for netflix for the last +20 years, that's pretty amazing! thank you for this video, great presentation!

[@souvikghosh8068]

I come back to this video once in a while to remind myself how cool the software engineering can be.

[@CodewithRSV]

This should be mandatory viewing for anyone working in a microservice architecture.

[@aceintheblackhole]

one of the best talks i've seen at any conference on any subject. i'm actually using it as a real-world resource for my system design interview preparation because it's very informative in many ways (especially for a beginner in distributed systems like me :) ). thank you for making this available!

[@cardboard3161]

Speaker: "Breathing is a miraculous act of bravery." 2020: "Challenge accepted."

[@Yusuf-ok5rk]

he is breathing in micro doses so that he can serve a continuous presentation

[@prakhargupta629]

Few years into corporate job and I had forgotten how many things are there in Computer Science. This talk brought me back to college.

[@antonvedel3028]

His analogies to the human body make great sense. Awesome video. Thank you.

[@somescams]

@Newtube Exactly.

[@madd5]

They do. But a regular monolithic application can also be analogous to a living organism.

[@thaituandat31]

I understand that Netflix is not only "NETFLIX" now. Thanks for your team, too. Amazing topic.

[@sirnawaz]

Brilliant talk! ... and the references to the various biological systems of the human body, throughout the talk, makes it even more interesting. It is amazing to see how various aspects of microservice architecture find similarities with other things created by the nature itself. On a microscopic level, we can draw analogies to the multiple-core, memory fences and cache-coherence as well.

[@bdpyne]

Really fantastic talk. He used analogies well, making abstract concepts more concrete. My most important takeaway was the evolving nature of architectures. Architectures are challenged both by external factors (customers) and internal factors (operations, customer support, etc.).

[@lexdemonica]

Such valuable insight from an articulate and knowledgeable insider. TYVM

[@jegtugado3743]

This talk is amazing. I'm far from being a system architect but this gives me a bigger perspective on how amazing systems like Netflix is built.

[@LesterFD]

copying and learning from nature is always something beautiful. natural sciences like physics, chemistry do it since centuries. It's a good time for computer science to learn from methods, that are applied for many thousands of years.

[@drxyd]

It's amazing how quickly the difficulty of engineering systems becomes with scale. My first instinct seeing the architecture was "too complex" but I've never worked on anything so large, would be awesome to see first hand.

[@sathvikvutukuri9179]

The way you explained microservices referring human body is mind blowing. Yes nature is the biggest Wikipedia or GitHub of everything. How much we learn or analyse or extract knowledge decides human evolution.

[@p3k1n0]

I could watch an entire season of this stuff. Way better than the standard Netflix products.

[@Innovate22]

The Alton Brown of software engineering nailed this presentation 👏👏👏

[@markmd9]

I just realized that on the globe we have more than 2 millions people that have interest in microservice architecture

[@cthackers]

"The structure of any system designed by an organization is isomorphic to the structure of the organization." Melving E.Conway

[@andrewmadge]

Great video! thank you for putting this on youtube

[@usun_current5786]

This is the most useful presentation I've ever seen regarding large scale platforms. Shared in my org, hopefully we'll lift a lot from here for future extensions.

[@clarkpeng8106]

what I've learnt important from this video is keeping perfecting your tools & architecture.

[@apurvsawant5703]

This is one of the best videos to get insights into micro services architecture.

[@bobross3305]

Thats crazy i never even thought about all this stuff before, because i use pirate bay.

[@kibiz0r]

AKA: Obfuscating Chaos - A Netflix Guide to Creating Dysfunctional Organizations Basically, a dev noticed: Hey, these two components could benefit from having isolated processes. And a manager said: Wait, could this also be used to stop Team A from complaining about consuming Team B's code? Thoughtful dev replied: Well, it depends on the specific components... Some components are fundamentally coupled, so they can't survive a partition anyway, and introducing one for the sake of team boundaries will just create more technical problems without solving the underlying social problems. Manager blinked twice, and after he rebooted he said: But they'll shut up about each others' code? Dev: I mean, technically. They'll just complain about API contracts instead -- Manager: Perfect. Dev: -- which is arguably worse, because as soon as you have a multistep process or something involving callbacks or dependency injection, you have a huge mess, and that's even if you can sidestep the CAP theorem for all of your use cases, which is a really big assump-- Manager: Yeah, yeah, whatever. So how soon can we do this?

[@thewebmaster1]

When adding a single table becomes a cross-functional problem, then the architecture suite of microservices is messily constructed, making backend administration of Data, correcting, adding or removing data, a complicated task instead of a simple 5 line operation. Middle Tier and Platform Services will one day cause the entropy of the Matrix, for a matrix it is. It is an Architecture that can one day fall like dominoes in many possible zero day scenarios

[@Larry21924]

This is absolute perfection. I recently read a similar book, and it was absolute perfection. "Mastering AWS: A Software Engineers Guide" by Nathan Vale

[@RonMartinJr]

I would love to just sit in a room with this guy and listen. Such a wealth of knowledge and experience.

[@OwaisAhmedMuhammad]

Simple and valueable. A very nice and practical talk on Microservices

[@Mike.e]

Excellent talk. Envious of your delivery! Thank you!

[@tobiasfuchs7016]

2:33 This visualization so beautiful and mesmerizing, and you immediately get the human body analogy. The whole presentation sure was hard work, and well worth it.

[@guseynismayylov1945]

it's not accurate though, it shows that microsevices are one directional, which is not true.

[@igrai]

very good and a very open talk, no generalities - really an insider's look into these challenges - awesome!

[@joechen2800]

Using organization as examples are really self-explanatory. Brilliant talk!

[@yao855]

Kubernetes was announced in July 2015, so I wonder why he said there's no good node orchestration tool out of the box at Netflix. Maybe they take extreme precautions when adopting new technology. Any insiders can confirm how they do it in 2020, did they build out in-house infra for that?

[@a0flj0]

I'mmalso curious what they are using. But, as a hands-on user of k8s, I think he's right about the no good orchestration tool. K8s is widespread and provides an abstraction layer for the actual cloud provider you are using, preventing lock-in. But it isn't actually that good. It has higher latencies than would be ideal, by design, caused by its model of constant sync of resource definitions to actual state. It keeps failing in unexplainable ways, even without unleashing a simian army upon it - luckily, rarely catastrophically. It provides low level support but no high level functionality for devops patterns that become increasingly expected/state of the art, such as blue-green deployments, canary deployments, continuous integration and continuous delivery. Its autoscaling capabilities are similarly limited, and autoscaling is also slow. Ops people are unecessarily exposed to many internal details. It's not outright bad, it's just quite far away from ideal, or even good. Hashicorp's Nomad is a much better solution, technically - it does less, but what if does it does better, and it lets you integrate things that solve additional problems easier. And it is more flexible in what it allows you to do - by far. These two aspects allow you to build infrastructure with a lot less cluttering, which is easier to analyze and reason abou, where coupling between different systems is lower. Unfortunately it's not as wide spread, and not as frequently available as a service. And it doesn't have a similarly strong marketing. But marketing alone doesn't make k8s good.

[@chungyuwaiki]

Great talk and hope i will understand much more years later.

[@yanikjayaram]

18:25 - hey guys does anyone know what software can produce this animated representation of a backend system? Would love to represent my own system this way to onboard other engineers easily.

[@ybenihime8757]

Awesome stuff, after watching this in 2020, i got myself thinking what changed in the architecture to this actual year.

[@VivekKartha]

Uncle Bob always talks with so much clarity. He has a deep understanding of all the topics he presents.

[@deepsue]

Thanks, this analogy, and learning is still happening and needs to be implemented in multiple applications/organization

[@himanipku22]

Thanks for reminding me I don't have to think about breathing.

[@chrisalister2297]

Built my system on MS ASP back in 2002. I was already using something more or less like micro-services and using ASPCache(like EVCache). Competitors would have issues/shut downs while mine systems kept trucking along with very little down time. Think of worker ants in colonies. All working towards a common goal.

[@dorcohen3522]

A disclaimer: if you don't have to scale different parts of your application independently, don't bother to go microservice. It won't solve any of your issues. Splitting an inherently coupled monolithic application will turn it into a distributed one, which is worse by any metric. Domains should be loosely coupled with or without microservice architecture. If your monolithic application is modular enough, there should be no problem to deploy it separately whenever needed.

[@jasonamaroreinert4499]

Great work. I know most of all of this content (whoopdie do I know), have been building microservices (via a standard reverse proxy/load balanced REST API-first architecture) since 2007, and yet this is still an amazing watch and important to keep instilling. That's because knowledge isn't a one-time thing, or a time-oriented thing at all but rather a means to solidifying a way. I love that this content is out there and presented this well because these stories are still happening out there even 15-20 years later!

[@anon_y_mousse]

When he mentions virtualization for dynamic resource allocation, is he talking about VM style virtualization, or more like the Erlang style of dynamically dispatching processes? I would assume the latter, but want to know how others interpreted it.

[@the.abhisheksinha]

nicely explained .. I wish Netflix gets all such videos on Netflix and show us how do they architect and design their mammoth system

[@Joe99]

On the topic of client libraries. A lot of recent talk argues that we should keep things DRY within a microservice bounded context, but not between bounded contexts. This introduces coupling and client libraries are one example. In the .NET world these shared client libraries can also lead to some nasty dependency versioning hell conflicts. I assume this issue is the same in the Java world, but I have no first hand experience. Oh yep, he mentions it at 21:07.

[@Walruz1000]

I read the same thing and am a bit confused by that as it seems to introduce a degree of coupling which in my limited studying seems to be something to avoid.

[@usun_current5786]

Yep, similar issues in Java world

[@not-high-on-life]

Fun thing is what was applicable to netflix(read usage of microservices) may not be applicable to your company/startup. A lot of software developers don't realise when something is applicable and start to put this shit everywhere they can reach... Exactly this is happening with crypto and microservices right now - hype oriented development, they hear buzz they look how cool it is and instantly start to put everywhere...

[@andreelyusef3235]

We need more of these kind of lectures. Spot on sir please do contribute more.

[@midnightsyntax3333]

Excellent presentation! The slides are super clear and specific. 5/5 kernels

[@conduit242]

"It started out innocently enough with Python..." So many things do...🤣

[@igatmatthew2283]

Innocuously*

[@lour.222]

scrapy comes to mind.....

[@parthyadav01]

@@lour.222 hahaha...

[@kim-uk1wv]

For shared client libraries. What about requiring each service to make their own wrapper classes/library that only exposes the parts they need. Then, to make it safer against changes, they also create their own unit-tests for their wrapper classes? Or if they use almost the entire client library they could just create their own set of unit-tests for the original library and skip the wrapper. When a shared client library is changed its test runner runs all the different wrapper library unit-tests. Then if anything brakes they can just contact the team responsible for the service owning the broken tests and work out a solution together.

[@a0flj0]

How do you solve API versioning and transitive dependency conflicts in the same library? He explained pretty well what the problems with shared libraries is.

[@frustratedmajority851]

The Theta Network is going to revolutionize how CDNs deliver content. It's a decentralized CDN network consisting of millions of user's smart devices acting as edge nodes that cache, encode, and stream video to those who need it... taking the work load off of the centralized CDNs (Netflix, KZbin, etc). This not only saves these centralized CDNs money in operational costs, but also improves the speed and quality of the video content by eliminating the "last mile problem".

[@ChidambaramVelayudham]

Absolutely Awesome. Love the way how Josh presented. Appreciate you for sharing the knowledge.

[@ahmadbaiazid976]

probably one of the best guides on youtube for microservice arch

[@prashanth0458]

I read the news of netflix introducing own bugs into their own system to make it more efficient long back, today I am watching how it is done. Good video with latest technological stuff like redundancy is good, bugs infection, support for multiple languages, region server failure routing etc..

[@ronfisher4751]

Very good stuff Josh you are a smart man and a great presenter of complex information

[@alvaromd3203]

presentation is great, hands down.

[@CoolHandMikeYT]

This is a really great talk to understand problems and solutions with microservice oriented archtecture at a large scale.

[@SartajHundal]

Turning Occam's Razor into a lecture. Well done.

[@raghavendrakrishnamurthy4041]

Awesome presentation!! Top class!

[@timriley1501]

How can anyone even give a thumbs down to this? Very interesting and informative video.

[@nabadeep09]

Just a funny observation, your video at 3:00 minutes has Corona virus structure in the slide for "Our Talk Today". This is the year 2020

[@Nampjg]

This man is genius... He explains things so well... It was enlightening to know that many of netflix's architecture optimizations are taken from Biology

[@ralphe9668]

wow thank you for this clear explanation of complex arcatechture. it confirms my theory of possible manipulation to the market demand rather convert manipulation for the highest bidder. just my opinion. what is your opinion?

[@ankitsorathiya2304]

Thanks Josh, It was an insightful talk, Thanks for sharing how netflix evolved over period of time.

[@dneary]

Interesting aroun 12:45 - updating the cache is, in his presentation, the responsibility of the service layer - I would have expected that to happen in the client library (AKA: check cache. If (miss), request result from service, store it in cache, and return it to user).

[@mattcargile]

Man! I really feel sad watching this like I missed the technology boat to be a part of something incredible! Like my life will never mean as much as this talk.

[@RobbykaGheo]

Well, this is a teory of experience, i think no one can present outspoken this without any extremely problem solved. If Netflix using Amazone then Amazone become rich because web services, i have self confidence for more focus diving into microservices world.

[@seanastro5549]

Just awesome... but wanna see the 2021 version now

[@srolesen]

I liked the 2017 version of netflix so much better, but thanks for this, fantastic presentation.

[@varunvora]

I'm wondering where they ended up with the debate on client libraries. I've certainly come across situations where I feel they are essential and some cases they are a burden.

[@LuisEDITS_KLK]

really good talk still in 2024 if you're new to microservices

[@DodaGarcia]

Surprised by the COVID cameo at 3:00 - always feels weird to see celebrities before fame

[@iwisdominheritance]

levelsetting, so that we have same vocabulary. challenges and solutions relationship between organisation and architecture

[@KentMarete]

Quality talk Josh. Question Josh Evans, what do you do when an entire region is compromised like succumbing into hacking activities? Do you kill the region fail over into the others to fix the vulnerability?

[@jaikantchandrakumaran3423]

Amazing talk! Great work Josh keep it up.

[@OneMinuteNotes]

Its one of the best story regarding microservices implementation. Cool

[@colloredbrothers]

10:30 anyone know what software was used to create that nice animated diagram? Edit: as said at the end of the presentation its called "visceral opensource"

[@sifiso5055]

vizceral

[@okie9025]

github.com/Netflix/vizceral

[@Master-ls2op]

KZbin SHOULD WATCH THIS!!!!! THE WAY THEY KEEP CRASHING ITS CRAZY!!!!!!!!!!!!!

[@andreeadianaratiu5091]

Great presentation and nice analogies.

[@Deepak_Tao]

Thanks Josh...very nice presentation!

[@udensijoshua9024]

Awesome delivery. This content is divine

[@sribalajirao]

Awesome post.. really worth viewing..

[@AsimMittal]

One of the best tech talks period

[@lepidoptera9337]

What do you do when your business plan is, essentially, the sale of CPU cycles? You offer products that use as many of them as possible to achieve as little as minimally required. Enter microservices. Genius.