Mastering Chaos - A Netflix Guide to Microservices

[@StingSting844]

Can't express how high quality this talk is. Wonder if he was a teacher before!

[@jordantaylor8549]

Yah I was wondering the same thing . Very top tier presentation and speaker

[@JamesJansson]

When you are the lead developer for a large org, it is your job to be an educator at a high level.

[@ThunderAppeal]

Youre full of shit. You have no idea of any of this.

[@yahorsinkevich4451]

I would say this is a perfect talk that describes how to writer microlyth and how NOT to design microservices based arch

[@StingSting844]

@@ThunderAppeal Yeah thats why I have come here to learn!

[@sundeepreddythirumuru5266]

One of the best talks I have seen on building systems of high scale, high availability, failure resilient and agility in undergoing change.

[@killoverflow]

One of the best microservices talk I have heared.

[@arthurkeech]

One of the best technical talks I've ever seen.

[@OliFubar]

I am sad I found this 3 years late, I've been developing microservices for the past 3+ years but I still learnt a lot from this. A huge belated thanks to you Josh!

[@sairampareek]

And I found this 3 years later you commented

[@ashimov1970]

1:51 "how amazing the human body is and how something as simple as an act of breathing or interacting with the world is actually a pretty miraculous thing. And it’s actually an act of bravery to a certain extent. There are so many forces in the world, so many allergens and bacterial infections and various things that can cause problems for us." - watching this on Feb 12, 2021 makes me admire this guy and the company he dedicated part of his miraculous life.

[@sandy.aggarwal]

Watched again 3 years later and it's still so relevant.

[@Inquisidor9408]

it doesn't get old

[@Patrickdaawsome]

Note to self: Write down summary starting from the 51:25. P.S. I watched the whole thing and absolutely loved it.

[@andrewferguson6901]

in case you didn't write it down where you could find it

[@tidal4774]

One of, if not the most comprehensive and analogous talks on the subject with relevant, real world stories from someone who has lived it. Thank you.

[@ErnstFluttert]

Great talk, which is still relevant in 2020! Especially the practical examples, and the solutions they applied. Loved the analogy with the human body. Thnx Josh!

[@martinmogusu]

This is a really informative talk on microservices, with real challenges and real solutions. The human body analogies make a lot of sense, there's a lot we can learn from nature...

[@kibiz0r]

AKA: Obfuscating Chaos - A Netflix Guide to Creating Dysfunctional Organizations Basically, a dev noticed: Hey, these two components could benefit from having isolated processes. And a manager said: Wait, could this also be used to stop Team A from complaining about consuming Team B's code? Thoughtful dev replied: Well, it depends on the specific components... Some components are fundamentally coupled, so they can't survive a partition anyway, and introducing one for the sake of team boundaries will just create more technical problems without solving the underlying social problems. Manager blinked twice, and after he rebooted he said: But they'll shut up about each others' code? Dev: I mean, technically. They'll just complain about API contracts instead -- Manager: Perfect. Dev: -- which is arguably worse, because as soon as you have a multistep process or something involving callbacks or dependency injection, you have a huge mess, and that's even if you can sidestep the CAP theorem for all of your use cases, which is a really big assump-- Manager: Yeah, yeah, whatever. So how soon can we do this?

[@UberOcelot]

As someone who doesn't deal with this side of architecture, this is a fantastic talk. Micro-services are just a division scheme, not really an architecture. To get things really running smoothly this provides some great insight to where the real work starts.

[@jonopens]

Such a fantastic metaphor to communicate the intent of microservices. As a younger programmer, that really resonated with me.

[@infoq]

If you are interested in microservices content you can check our collection on InfoQ. www.infoq.com/microservices

[@souvikghosh8068]

I come back to this video once in a while to remind myself how cool the software engineering can be.

[@dorcohen3522]

A disclaimer: if you don't have to scale different parts of your application independently, don't bother to go microservice. It won't solve any of your issues. Splitting an inherently coupled monolithic application will turn it into a distributed one, which is worse by any metric. Domains should be loosely coupled with or without microservice architecture. If your monolithic application is modular enough, there should be no problem to deploy it separately whenever needed.

[@gashinamu]

One of the best tech talks I've ever watched.

[@infoq]

Happy you like it. You can find more on www.infoq.com/

[@CodewithRSV]

This should be mandatory viewing for anyone working in a microservice architecture.

[@prakhargupta629]

Few years into corporate job and I had forgotten how many things are there in Computer Science. This talk brought me back to college.

[@cthackers]

"The structure of any system designed by an organization is isomorphic to the structure of the organization." Melving E.Conway

[@jidnyasadhavale6664]

So Informative. Tech talks are the best way to get insight into how huge applications are built, complexities involved, challenges and solutions.

[@a0flj0]

Actually, no. What you get, mostly, from tech talks, is various tech people advertising their pet technology, in a way that appeals to other tech people. If you really want to really get into big systems, find work on one, and do thorough research - tech talks are a good start, but often leave important things out, and only show you the pictures of the car's body, without teaching you to work on the engine. To learn about cars, you go to grease monkey school, not to car shows. The reason for our industry being insanely hype driven is probably at least partially that people expect to actually learn something from tech talks, instead of considering them simple pointers to start research from.

[@Innovate22]

The Alton Brown of software engineering nailed this presentation 👏👏👏

[@jegtugado3743]

This talk is amazing. I'm far from being a system architect but this gives me a bigger perspective on how amazing systems like Netflix is built.

[@clarkpeng8106]

what I've learnt important from this video is keeping perfecting your tools & architecture.

[@bdpyne]

Really fantastic talk. He used analogies well, making abstract concepts more concrete. My most important takeaway was the evolving nature of architectures. Architectures are challenged both by external factors (customers) and internal factors (operations, customer support, etc.).

[@apurvsawant5703]

This is one of the best videos to get insights into micro services architecture.

[@markmd9]

I just realized that on the globe we have more than 2 millions people that have interest in microservice architecture

[@sirnawaz]

Brilliant talk! ... and the references to the various biological systems of the human body, throughout the talk, makes it even more interesting. It is amazing to see how various aspects of microservice architecture find similarities with other things created by the nature itself. On a microscopic level, we can draw analogies to the multiple-core, memory fences and cache-coherence as well.

[@LesterFD]

copying and learning from nature is always something beautiful. natural sciences like physics, chemistry do it since centuries. It's a good time for computer science to learn from methods, that are applied for many thousands of years.

[@OrdinaryFemmale]

this beast is *ONLY* working for netflix for the last +20 years, that's pretty amazing! thank you for this video, great presentation!

[@lexdemonica]

Such valuable insight from an articulate and knowledgeable insider. TYVM

[@thaituandat31]

I understand that Netflix is not only "NETFLIX" now. Thanks for your team, too. Amazing topic.

[@antonvedel3028]

His analogies to the human body make great sense. Awesome video. Thank you.

[@somescams]

@Newtube Exactly.

[@madd5]

They do. But a regular monolithic application can also be analogous to a living organism.

[@p3k1n0]

I could watch an entire season of this stuff. Way better than the standard Netflix products.

[@aceintheblackhole]

one of the best talks i've seen at any conference on any subject. i'm actually using it as a real-world resource for my system design interview preparation because it's very informative in many ways (especially for a beginner in distributed systems like me :) ). thank you for making this available!

[@drxyd]

It's amazing how quickly the difficulty of engineering systems becomes with scale. My first instinct seeing the architecture was "too complex" but I've never worked on anything so large, would be awesome to see first hand.

[@joechen2800]

Using organization as examples are really self-explanatory. Brilliant talk!

[@sathvikvutukuri9179]

The way you explained microservices referring human body is mind blowing. Yes nature is the biggest Wikipedia or GitHub of everything. How much we learn or analyse or extract knowledge decides human evolution.

[@himanipku22]

Thanks for reminding me I don't have to think about breathing.

[@the.abhisheksinha]

nicely explained .. I wish Netflix gets all such videos on Netflix and show us how do they architect and design their mammoth system

[@usun_current5786]

This is the most useful presentation I've ever seen regarding large scale platforms. Shared in my org, hopefully we'll lift a lot from here for future extensions.

[@RonMartinJr]

I would love to just sit in a room with this guy and listen. Such a wealth of knowledge and experience.

[@thewebmaster1]

When adding a single table becomes a cross-functional problem, then the architecture suite of microservices is messily constructed, making backend administration of Data, correcting, adding or removing data, a complicated task instead of a simple 5 line operation. Middle Tier and Platform Services will one day cause the entropy of the Matrix, for a matrix it is. It is an Architecture that can one day fall like dominoes in many possible zero day scenarios

[@deepsue]

Thanks, this analogy, and learning is still happening and needs to be implemented in multiple applications/organization

[@OwaisAhmedMuhammad]

Simple and valueable. A very nice and practical talk on Microservices

[@igrai]

very good and a very open talk, no generalities - really an insider's look into these challenges - awesome!

[@ybenihime8757]

Awesome stuff, after watching this in 2020, i got myself thinking what changed in the architecture to this actual year.

[@Mike.e]

Excellent talk. Envious of your delivery! Thank you!

[@chungyuwaiki]

Great talk and hope i will understand much more years later.

[@andrewmadge]

Great video! thank you for putting this on youtube

[@andreelyusef3235]

We need more of these kind of lectures. Spot on sir please do contribute more.

[@VivekKartha]

Uncle Bob always talks with so much clarity. He has a deep understanding of all the topics he presents.

[@bobross3305]

Thats crazy i never even thought about all this stuff before, because i use pirate bay.

[@ahmadbaiazid976]

probably one of the best guides on youtube for microservice arch

[@CoolHandMikeYT]

This is a really great talk to understand problems and solutions with microservice oriented archtecture at a large scale.

[@Larry21924]

This is absolute perfection. I recently read a similar book, and it was absolute perfection. "Mastering AWS: A Software Engineers Guide" by Nathan Vale

[@not-high-on-life]

Fun thing is what was applicable to netflix(read usage of microservices) may not be applicable to your company/startup. A lot of software developers don't realise when something is applicable and start to put this shit everywhere they can reach... Exactly this is happening with crypto and microservices right now - hype oriented development, they hear buzz they look how cool it is and instantly start to put everywhere...

[@SartajHundal]

Turning Occam's Razor into a lecture. Well done.

[@kim-uk1wv]

For shared client libraries. What about requiring each service to make their own wrapper classes/library that only exposes the parts they need. Then, to make it safer against changes, they also create their own unit-tests for their wrapper classes? Or if they use almost the entire client library they could just create their own set of unit-tests for the original library and skip the wrapper. When a shared client library is changed its test runner runs all the different wrapper library unit-tests. Then if anything brakes they can just contact the team responsible for the service owning the broken tests and work out a solution together.

[@a0flj0]

How do you solve API versioning and transitive dependency conflicts in the same library? He explained pretty well what the problems with shared libraries is.

[@cardboard3161]

Speaker: "Breathing is a miraculous act of bravery." 2020: "Challenge accepted."

[@Yusuf-ok5rk]

he is breathing in micro doses so that he can serve a continuous presentation

[@Nampjg]

This man is genius... He explains things so well... It was enlightening to know that many of netflix's architecture optimizations are taken from Biology

[@timriley1501]

How can anyone even give a thumbs down to this? Very interesting and informative video.

[@prashanth0458]

I read the news of netflix introducing own bugs into their own system to make it more efficient long back, today I am watching how it is done. Good video with latest technological stuff like redundancy is good, bugs infection, support for multiple languages, region server failure routing etc..

[@chrisalister2297]

Built my system on MS ASP back in 2002. I was already using something more or less like micro-services and using ASPCache(like EVCache). Competitors would have issues/shut downs while mine systems kept trucking along with very little down time. Think of worker ants in colonies. All working towards a common goal.

[@ronfisher4751]

Very good stuff Josh you are a smart man and a great presenter of complex information

[@tobiasfuchs7016]

2:33 This visualization so beautiful and mesmerizing, and you immediately get the human body analogy. The whole presentation sure was hard work, and well worth it.

[@guseynismayylov1945]

it's not accurate though, it shows that microsevices are one directional, which is not true.

[@Joe99]

On the topic of client libraries. A lot of recent talk argues that we should keep things DRY within a microservice bounded context, but not between bounded contexts. This introduces coupling and client libraries are one example. In the .NET world these shared client libraries can also lead to some nasty dependency versioning hell conflicts. I assume this issue is the same in the Java world, but I have no first hand experience. Oh yep, he mentions it at 21:07.

[@Walruz1000]

I read the same thing and am a bit confused by that as it seems to introduce a degree of coupling which in my limited studying seems to be something to avoid.

[@usun_current5786]

Yep, similar issues in Java world

[@jasonamaroreinert4499]

Great work. I know most of all of this content (whoopdie do I know), have been building microservices (via a standard reverse proxy/load balanced REST API-first architecture) since 2007, and yet this is still an amazing watch and important to keep instilling. That's because knowledge isn't a one-time thing, or a time-oriented thing at all but rather a means to solidifying a way. I love that this content is out there and presented this well because these stories are still happening out there even 15-20 years later!

[@srolesen]

I liked the 2017 version of netflix so much better, but thanks for this, fantastic presentation.

[@midnightsyntax3333]

Excellent presentation! The slides are super clear and specific. 5/5 kernels

[@OneMinuteNotes]

Its one of the best story regarding microservices implementation. Cool

[@ChidambaramVelayudham]

Absolutely Awesome. Love the way how Josh presented. Appreciate you for sharing the knowledge.

[@conduit242]

"It started out innocently enough with Python..." So many things do...🤣

[@igatmatthew2283]

Innocuously*

[@lour.222]

scrapy comes to mind.....

[@parthyadav01]

@@lour.222 hahaha...

[@mattcargile]

Man! I really feel sad watching this like I missed the technology boat to be a part of something incredible! Like my life will never mean as much as this talk.

[@kevinparker825]

We moved away from micro-services and back to a monolithic code base. For our project size and advanced features it just wasn't working, was costing millions in developer time and lost revenue. The added complexity of not being able to call functions within code, the need resolve things through a tunnel and other micro service paradigms was excessive...

[@AsimMittal]

One of the best tech talks period

[@lepidoptera9337]

What do you do when your business plan is, essentially, the sale of CPU cycles? You offer products that use as many of them as possible to achieve as little as minimally required. Enter microservices. Genius.

[@alvaromd3203]

presentation is great, hands down.

[@LuisEDITS_KLK]

really good talk still in 2024 if you're new to microservices

[@yao855]

Kubernetes was announced in July 2015, so I wonder why he said there's no good node orchestration tool out of the box at Netflix. Maybe they take extreme precautions when adopting new technology. Any insiders can confirm how they do it in 2020, did they build out in-house infra for that?

[@a0flj0]

I'mmalso curious what they are using. But, as a hands-on user of k8s, I think he's right about the no good orchestration tool. K8s is widespread and provides an abstraction layer for the actual cloud provider you are using, preventing lock-in. But it isn't actually that good. It has higher latencies than would be ideal, by design, caused by its model of constant sync of resource definitions to actual state. It keeps failing in unexplainable ways, even without unleashing a simian army upon it - luckily, rarely catastrophically. It provides low level support but no high level functionality for devops patterns that become increasingly expected/state of the art, such as blue-green deployments, canary deployments, continuous integration and continuous delivery. Its autoscaling capabilities are similarly limited, and autoscaling is also slow. Ops people are unecessarily exposed to many internal details. It's not outright bad, it's just quite far away from ideal, or even good. Hashicorp's Nomad is a much better solution, technically - it does less, but what if does it does better, and it lets you integrate things that solve additional problems easier. And it is more flexible in what it allows you to do - by far. These two aspects allow you to build infrastructure with a lot less cluttering, which is easier to analyze and reason abou, where coupling between different systems is lower. Unfortunately it's not as wide spread, and not as frequently available as a service. And it doesn't have a similarly strong marketing. But marketing alone doesn't make k8s good.

[@raghavendrakrishnamurthy4041]

Awesome presentation!! Top class!

[@frustratedmajority851]

The Theta Network is going to revolutionize how CDNs deliver content. It's a decentralized CDN network consisting of millions of user's smart devices acting as edge nodes that cache, encode, and stream video to those who need it... taking the work load off of the centralized CDNs (Netflix, KZbin, etc). This not only saves these centralized CDNs money in operational costs, but also improves the speed and quality of the video content by eliminating the "last mile problem".

[@ankitsorathiya2304]

Thanks Josh, It was an insightful talk, Thanks for sharing how netflix evolved over period of time.

[@seanastro5549]

Just awesome... but wanna see the 2021 version now

[@paolodisintegra]

And here i am trying to understand how to create a "User" microservice with node lol

[@anonymous4711_]

How far along are you? Have you built some nice stuff meanwhile?

[@ilin76bb]

hi paolo, 2 years later, did you managed to create the user microservice ?

[@paolodisintegra]

@@ilin76bb I've managed to create the user microservice for the company i was working for at the time but i end up giving up on developing others for the remaining entities. I discovered it's better to build a monolith to avoid technical debt and time wasting integrations and then split it later on when scaling is needed. It's actually overkill to start a project with microservices specially if you are the only developer on the project. It's just too much work for one developer only...

[@feminist098]

@@paolodisintegra agreed

[@DJ_POOP_IT_OUT_FEAT_LIL_WiiWii]

@@paolodisintegra wow great insight there, thanks for sharing

[@Master-ls2op]

KZbin SHOULD WATCH THIS!!!!! THE WAY THEY KEEP CRASHING ITS CRAZY!!!!!!!!!!!!!

[@wartem]

good stuff

[@Sugmatron]

I would take this talk more seriously if Netflix wasn't a wildly chaotic mess where you need to log in on a specific platform to modify your subtitle options.

[@cinemarob1]

2:04: "There are so many forces in the world, so many...bacterial infections and various things that can cause problems for us" - Ahead of its time

[@DodaGarcia]

And then bam, surprised COVID image at 3:00 lmao

[@cinemarob1]

@@DodaGarcia Loom'nati!

[@soumyachatterjee4135]

16:29 The vaccine doses :P This went to spooky level

[@ralphe9668]

wow thank you for this clear explanation of complex arcatechture. it confirms my theory of possible manipulation to the market demand rather convert manipulation for the highest bidder. just my opinion. what is your opinion?

[@Omikoshi78]

I was gonna look for a movie to watch but this was far too interesting. An hour well spent.

[@Deepak_Tao]

Thanks Josh...very nice presentation!

[@andreeadianaratiu5091]

Great presentation and nice analogies.

[@starkest]

I feel bad for Yahoo

[@RobbykaGheo]

Well, this is a teory of experience, i think no one can present outspoken this without any extremely problem solved. If Netflix using Amazone then Amazone become rich because web services, i have self confidence for more focus diving into microservices world.

[@sribalajirao]

Awesome post.. really worth viewing..

[@CalamityMIC]

32:17 When you're a sports cameraman doing a conference for some side cash

[@letscifucan]

This Lecture was awesome.

[@anon_y_mousse]

When he mentions virtualization for dynamic resource allocation, is he talking about VM style virtualization, or more like the Erlang style of dynamically dispatching processes? I would assume the latter, but want to know how others interpreted it.

[@JiaxinLiu]

Hystrix is very cool! Thanks for sharing!