Microservices Gone Wrong at DoorDash

[@NeetCodeIO]

A principal engineer and one of the original authors of the blog post from DoorDash emailed me with a couple corrections: """ 1. cascading failure: the point here is not that an error or latency experience in an RPC call graph is experienced by all upstream callers. Rather it's that one thing being broken or slow ends up breaking a totally other thing, usually in a different way. As an example, let's say a database gets slow and it's used in a call graph by multiple services. So like A->B->db, C->B->db. By cascading failure we mean that because db is slow, A or B develop a NEW kind of failure, like they run out of memory or threads or something like that. 2. death spiral: scaling out is often a part of it, but the real issue is how to handle an overload condition. If there's no central coordinator, then each node needs to make this determination around overload individually. """ I will try to make a follow-up video covering these. Written version: neetcode.io/newsletter System Design Playlist: kzbin.info/www/bejne/onfSnnSPqsSInZI

[@funkdefied1]

I love this. It’s not just reading and reacting. It’s original content

[@robertluong3024]

Primeagen and Theo in shambles

[@JegErN0rsk]

Primeage is good at bringing his own thoughts and knowledge

[@imdeadserious6102]

People who read are curators and add their own thoughts. You probably wouldn't read all the articles yourself. And now you get 3 layers of input, yourself, KZbin reader, and the article.

[@TylerFields-o8p]

So tired of Prim and Theo... hardly ever anything good anymore. We get it, you have opinions, but i want to know what the title of your video is about, you know?f

[@aveonhx]

@@robertluong3024 Theo doesn't even put any efforts into making such videos, sits and yap like a maniac.

[@arniie5288]

I love these kinds of system design videos, would love to see more coming!

[@hellowill]

monoliths can scale (performance wise) too. microservices are more about scaling deployments and teams.

[@jonas-froeller]

I like monoliths. I don‘t like having to type something for every new microservice.

[@andreiga76]

It depends on how uniform the processes are, if you also have some high CPU/GPU usage processes it's better to put those on different services (not necessary microservices) so you can use specific nodes for them, otherwise you would have to scale the infrastructure with a lot of high cost CPU/GPU nodes to satisfy both simple operations and compute intensive operations (good luck telling that to managers).

[@illu1na]

nah there is a limit to which monolith can scale. Obvious reasons are database connection limit and load to the master database.

[@kevincrotty826]

Pedantic nit pick: a negative feedback loop is self damping (it negates the output, then feeds it back in). You have described a positive feedback loop (the input begets even more of the same as output) that has an undesirable outcome.

[@Flowy653]

pedantic or not, it ruined whole video for me

[@rednafi]

@@Flowy653 quit being so overrly dramatic. Write your own content then.

[@pubcollize]

He clearly said "negative feedback-loop", not "negative-feedback loop"

[@nthmost]

This is exactly why messaging queuing became popular in the 2010s, to enable the decoupling of these services so as to avoid bottlenecks and timeouts. This isn't always possible (read: appropriate for the use case), but important for software architects to keep in mind as a potential solution.

[@Hidacal]

Its more of a hack than a solution. Its popular because it solves the immediate problem, but it also creates a integration hell and does absolutely nothing to decouple services, only shoves problems under a rug for someone to deal with later. Coupling virtually always exists do to business requirements, not architecture. Changing how it looks doesnt change the fact that it exists.

[@purdysanchez]

Most of the large ecosystems of micro services I've gotten to dive into have ironically been way more expensive and way slower than just building regular APIs with reasonable service boundaries. Then in order to make the microservice driven website appear performant, they spend a ton of money on monolithic caching layers between the client and the microservice APIs.

[@HairyCheese]

I've seen it where we need to separate a production back end away from a client facing the front end. I'd put the micro service against the client facing front end, some risk of stale data, but using event streaming from the back end should scale well. Whaddya think?

[@mjrmjr4]

Hi, I work at DoorDash on this kind of thing. A lot has changed since we wrote that blog post. I'd be happy to talk to you about the way we are thinking about microservices now if that's something you'd be interested in.

[@NeetCodeIO]

Thanks, just emailed you!

[@mjrmjr4]

@@NeetCodeIO Alright, let's make this happen.

[@didyoustealmyfood8729]

Let them cook

[@dukedarkwolfe]

get out matt

[@caspera3193]

Do not build distributed monoliths. Avoid temporal coupling. And if you can, avoid microservices unless you cannot run from them anymore.

[@manojBadam]

What do you mean by Temporal coupling?

[@jonlogrippo8312]

Not a programmer. This is so over my head I thought I was on a sci-fi channel for a sec

[@9e7exkbzvwpf7c]

@@manojBadamto be clear I disagree with them but they mean building sync chains of calls, that your microservices should be handling async events.

[@tomasbajarunas6416]

When does microservice architecture stops being a microservice architecture and becomes a distributed monolith one? When does distributed monolith stops being one and becomes modular monolith? I tried to look for good source to read more about, all those three, being compared, I think I understand microservices, I understand modular monolith, but recently heard about distributed monolith, and am wondering now what is that all about, and maybe my understanding of all those three is not as good as I thought it was.

[@palmwineguy]

@@manojBadam it sounded like some gandolf stuff

[@hapaise2924]

I love this type for content, u get to learn soo much concepts and overview of systems and understand whats going on. keep this up

[@ngominhsaoS2]

2:30 strictly speaking, this is not microservices architecture, it's more like service-orient architecture.

[@sidasdf]

Interesting that these are exactly the problems faced in networking. The retry storm issues / prevention techniques are exactly the same as the problems / solutions that one faces with retransmission logic in TCP

[@KrishnaList]

I will just say Big WOW. High quality information on this channel. I would say please make more videos on microservice use cases like this one.I did not find such a clear explanation in KZbin.

[@alexclark6777]

So it sounds like DoorDash just made a complete hash of their microservice architecture and did every tropey-microservice misstep in the book. I'm not even pro-microservice but their failure isn't exactly a detractor for microservices, it's just another solid example of a company adopting an architecture without properly understanding it.

[@gaulcore]

This is a great presentation. Love this format. Keep it up!

[@bananasmileclub5528]

This is pretty good content. I have never really liked system design kind of questions/reflections but this def brought it in a new light for me. Thank you for that.

[@cameronosborne7405]

Death spiral: multiple services spin up depending on metrics read by horizontal pod autoscaler in K8s, or KEDA metrics. Time to spin up a new pod is pretty low, you would always have enough instances running to handle failure tolerance. If there is only one node left, your service should have a “prefetch count” to only fetch a certain amount of messages in the queue(s) assigned to it before resting any more messages.

[@nark4837]

regarding the death spiral, wouldn't it also be wise to have some autoscaling rule on if X many deaths within Y minutes, scale up by Z nodes? i feel like this would mitigate this issue also tbh, i feel like if they're using gRPC for every service anyway, they had that coming, use asynchrony where possible, e.g., MQs, rather than just jamming requests down a microservices throat when it doesn't have the capacity to handle them

[@someguyO2W]

What's the point then?

[@cameronosborne7405]

Retry storm: solved by using retries at the broker level such as rabbitmq, or service mesh layer such as istio, or workflow orchestrator level. Also, if services are connected by a broker or service mesh, the load balance between service instances is automatic and multiple services spin up depending on the load.

[@cameronosborne7405]

Cascading failure: solved by workflow orchestration microservice that speaks to multiple services in a stepwise sequence with failure and compensation rollbacks

[@tomasbajarunas6416]

To address scaling, instead of scheduled one, or based on load, another option could be to have a combination: scale on predicted load times, but in addition, scale early - your combined server usage is reached 50% - spawn +10% of current number of instances. You have a sudden spike, server loads reached 75%, spawn +2 right away. Various techniques could be implemented with some watchdog observing servers and proactively managing microservice instances. Spawning services shouldn't actually take that long.. well, depends.. on AWS, there is a difference between how ECS are spawned: Fargate vs EC2 instances, and in case ECS are spawned in EC2 instances, then it would take time to get a new EC2 instance, spin it, etc.. Fargate is more expensive, but is faster to get it running.. And here we have it - scheduled scale: use EC2, you need an immediate few instances - spawn Fargate, while spawning instances on EC2 as well, thus Fargate satisfies an initial load fast, and could be killed first, as soon as ECS services on EC2 come alive, and load goes down bellow certain threshold, and we start killing cluster instances - always downsize Fargate ones first. There is additional overhead in tems of how much you need to pay attention to dev-ops side of this architecture, and I think the real issue for DoorDash isn't in not having brainpower, or because of lack pf experience, but rather not willing to be proactive and waiting till it really becomes a necessity to address that technical debt. Future me problem, and taking a least resistance path, even though those scenarios are pretty clear. BTW, good video, we need more of those so that companies be aware and stop pushing microservices, which in most cases is an unnecessary complicated architecture: when you need, you need it, but in most cases you don't. It's not a silver bullet, true horizontal scaling comes at price.. At the moment, microservices are overhyped, and it's a buzzword, and it seems that majority of solutions should implement a monolith modular architecture rather than a microservices one.

[@debkanchan]

First mistake was making "microservices" that directly depend on one another and hence coupled(which is what microservices shouldn't be).

[@qapplor]

so how can they work together, when there can be no depedency at all?

[@jett896]

If a service can function without other services....why are those other services even necessary? If there was no dependency, the service wouldn't exist.

[@highdynamic]

Couple thoughts: 1. the need for load shedding could usually be a sign that your microservices are not that "micro" after all. If in one service you end up with very different load patterns on certain endpoints, you should consider splitting there to have these patterns scale independently of each other (not necessarily a full different "service" but at least shaved off and put on their own "infrastructure" within that "service") 2. auto-scaling: predictive scaling is amazing if you have consistent historical patterns or can make very good predictions, but you can also go a long way if you just really fine-tune your auto-scaling rules: give more "breathing space" for the system in overall and run it at a lower overall resource utilization, so it won't be so "tight". You don't need to do this necessarily by having to pay for extra unused resources constantly, what I mean is that you can be more generous with the resources given when scaling. Give more and stronger instances to the pool at the first scale trigger kicking in, kind of be more "aggressive at first sight" and scale out AND up at the same time in a pre-emptive fashion (granularity of the settings you have ofc may differ based on what cloud service you are using, not all have super detailed options). Depends of course on each case and business, but many companies are just using built in default settings or very basic approach, and by giving a higher initial "scale response" and really fine-tune the scaling triggers (and cooldowns) to their own product, life would be a lot easier and less issues with spikes. (not saying DoorDash did or did not do anything, just in general I've often seen auto-scaling configurations being underutilized and too vague or universal)

[@hellowill]

network latency isn't that bad in my experience. The main latency is the serialisation/deserialisation overhead.

[@rednafi]

What language are you using for serialization and deserialization? In my experience, serde load accounts for less than 10% of network overhead in a real system.

[@dealloc]

Depends on your structures, size of the data and what you are de/serializing and what the format you send through. Binary can be more efficient, whereas string representations (i.e. JSON) can be more flexible. Protobufs are more complicated, but aims to provide both flexibility and efficiency (albeit with additional complexity).

[@SETHthegodofchaos]

any thoughts on gRPC or other binary formats? over json or xml?

[@yaboy7120]

good stuff man you got a new sub! clicked out of curiosity expecting to click off but by the end I was disappointed that it was over lol. You break things down well

[@taiyesalami6014]

More of this type of videos please. Really edicational

[@RadoMich]

Wow... just the nuts and bolts, instead of usual performance-scalability-cost-saving-observability-mumbo-jumbo you can easily find on other channels.

[@DsBoyan]

Can't believe I lived to hear Java and performance used in the same sentence 😆

[@mkres]

yeah, he clearly didn't mean it, replacing python with java for performance reasons is still not a thing :D

[@laughingvampire7555]

the thing is that is very difficult to identify the boundaries for the microservice, the independent unit of execution.

[@Dave.Wattz100]

We use microservices at company that I work at and we categorize our services. One category is coordinator service, such service is responsible for making request to all other microservices that are necesarry for a certain task. No other microservice is not allowed to make requests to other microservices. I think it kinda solves this problem ?

[@BigBrotha3459]

Like a mediator right?

[@BlTemplar]

Your solution solves some problems but brings new ones. Imagine you have a service A which makes a call to service B. The client accesses the service A directly. But you have a coordinator so you have service C which makes calls to both A and B. Two requests instead of one and the coordinator becomes an additional point of failure in your system. Doordash problem wasn't that microservices call other microservices but that there was not enough protection against outages and no graceful degradation.

[@varshard0]

Now the coordinator becomes a single point of failure with added latency. imo, it's better to implement other microservice patterns like circuit breakers, bulkhead and etc. Introducing some events driven in some areas where eventual consistency is allowed is also not a bad idea.

[@adambickford8720]

Google 'choreography vs orchestration'; these are known tradeoffs.

[@milliepards96]

Now that I’m actually working as a SWE, I find these videos so much more interesting than Leetcode puzzles.

[@dontdoit6986]

TBH for years the world was designing micro-services incorrectly and it was an uphill battle to correctly implement them. And then, over the crest, now there are attacks on micro-services. 😅

[@johnboy14]

In most cases they’re completely unnecessary. I don’t buy this argument that we must break our architecture so more people can work on it. Linux doesn’t do this and there's millions of lines of code in there. If your code base has proper separation of concerns and nice points of abstraction, then multiple people can work on the same codebase. Ive seen it work. I favour libraries over microservices and I'd still write versions in multiple libraries than turn one codebase into 4 different services.

[@adityaanuragi6916]

2:35 as a red-green colour blind person this is hard to read for me Slight suggestion - both healthy and unhealthy service have same pattern of lines, perhaps change just one of them

[@NeetCodeIO]

Good suggestion and sorry for the issue, I should've known that. Will keep it in mind for future videos.

[@jrdtechnologies]

Perhaps doordash could have benefited from architecting their databases utilizing database sharding design patterns to minimize down time and to improve availability? Or minimally reduce the severity of the impact by isolating the database maintenance issue to a smaller area perhaps

[@Patmorgan235Us]

They did this, and there's several entries on their engineering blog about their DB engineering

[@montramedia]

as far as the death spiral why not just lower the traffic/usage threshold ?

[@norunners_]

I’m surprised health checks were not discussed on the topic of availability. It’s one way for the load balancing to detect unhealthy hosts without relying on an unhealthy host to monitor its own resources. It’s often best to remove an over burdened host from an LB, thus failing fast can lead to faster mitigations and recovery times.

[@hentie5371]

failure 3 is just a simple problem and it is also not a problem unique to microservice architecture. CB exist for a reason, its best to fail some requests than to fail the service along with every requests coming into it.

[@rogers2934]

Surprised they didn't consider choreography for this problem. In our experience, a blend of choreography and orchestration has worked well for most of our use cases. However, it's possible that it just doesn't suit their specific requirements.

[@domingos_m]

In South Africa load shedding has a whole different meaning

[@NathanielBabalola]

How do you do predictive auto scaling when your app is used in multiple countries in different time zones ?

[@ravi72munde]

not sure how Doordash does it but most companies will try to recreate the whole stack in a different region so they pretty much act independently. But something for say within US but different time zones it still is very predictable(eg: Doordash will probably see surge of orders at 12 pm ET and 12 pm PT).

[@NathanielBabalola]

@@ravi72munde okay fair enough, I get you. Thanks

[@ChrisCebelenski]

"No Silver Bullet" - as valid today with microservices and honestly any other architectural patterns. And it's an easy to get into trouble like latency and poor optimization and circular jerk with monoliths as with microservices. But orgs get into trouble faster with microservices when there is no design or coordination - and the problem is only worse with poor testing regimes and bad tooling - it takes more maturity or process with microservices. And poor developers write poor code no matter the paradigm. With good tooling and deployment automation (GitOps!) and containerization (K8s) and automated testing, all boats are raised by the tide.

[@funkdefied1]

A lot of the strategies used to mitigate these orchestration issues seem pretty complex. What’s the best way to implement these strategies at scale? Does every microservice implement their own request IO strategy? Or is there a scalable solution external to the microservices?

[@Viviko]

What’s with all the waiting? Isn’t it better to just replicate data across services and make use of eventual consistency? Like, instead of waiting for something to complete, just fire an event and have the interested service listen and update its state… eventually?

[@avrelyy]

Good explanation of issue with microservices. Thank you

[@fudgeyfoot419]

It seems to me like a majority of these things apply to a monolith service too. How does a death spiral only apply to a microservice architecture and not a monolith service? As a matter of fact, I would argue that a microservice architecture would be MORE helpful in a death spiral situation since it is isolated to one of your many microservices.

[@ecereto]

I agree with the conclusion that microservices only help when the amount of people working on the same code are in the thousands. Even then you probably could make do with better architecture. I'm having flashbacks of the time I tried to right a small poece of software usong microservices and was so stuck on figuring out gRPC and message passing that i never even started working on my idea before i gave up.

[@jamiebrs1]

Makes sense they had that spike in 2020 when the pandemic hit and most people were staying home.

[@zacki5663]

this was a great video. i can tell you we use to run an interview where we'd ask what failure modes there are at scale and the mitigations, and very few people could answer past "thundering herd" and "exponential backoff"

[@10000rah]

I don’t see the microservices is the problem here, it how they splitting the monolith to microservice is the problem

[@SETHthegodofchaos]

10:30 Hm, why would an overloaded node crash tho?

[@VaibhavShewale]

damn, i think doordash have created their own tool for this problem and not teling about it but incoming months they will share how they created the tool to tackle the issue and it solved the problem and will probably will donate it to open source community or sell that tool

[@VidyaBhandary]

Another great explanation !!! Thank you !!!

[@mohamed44324]

I am really enjoying this content. we need more of this

[@gigh-ced-joun-nand]

Did Uber not use event-based architecture? Meaning most service to service communication is a fire-and-forget event that doesn't require a response.

[@DaveOzoalor]

Whats the advantage of this?

[@mikhail-t]

Thanks for great video! Am I right that Failure 4 (Metastable failures) is just a consequence of Failure 3 (or similar) where system cannot recover to stable state by itself? Trying to understand why it was separared into dedicated failure type. Btw, during this video I was waiting for any mention of backpressure ...

[@mehuljain1991]

Very Interesting! Make more like this.

[@xeridea]

Cascading failure, seems they was hyper aggressive on assuming latency was a failure, and timeout thresholds were set way too low. Auto scaling, why would they take down nodes that were being replaced, and why jump to conclusion to aggressively replace stuff all the time? Just scale up and only replace if there are actual failures, this would significantly mitigate the negative feedback loop. And high latency shouldn't be considered an error.

[@JegErN0rsk]

More if this. Great video!

[@silversurfer1967]

what software did u use to create the sketch ?

[@DaRealCodeBlack]

They should have gone with erlang

[@NurHossainRidoy]

what books or articles or websites do you follow to learn system design or architechture.

[@johnw.8782]

I appreciate your channel and the information you're trying to get across, but what is your actual real world experience with any of these types of architectures?

[@andreiga76]

I'm not sure I understand the "Payment" vs "Image" example, isn't that the trigger to "request" separating the components in different services so they can be scaled and maintained separately? In my view the client (calling service or whatever) should be in charge to say if something is critical or not and how to manage the fails because it knows the business logic needs, not the called service (especially if it is a microservice).

[@FreeDomSy-nk9ue]

Are these videos part of a live stream or something? Where can I watch?

[@SamSedighian]

Actually engineering is one of our smaller teams so you are correct. I would guess 10% of our employees are engineering

[@iAPX432]

Obvious solution are prioritized (and timing out) queues instead microservices direct calls.

[@0xAC]

All this talk about microservices that those are needed to achieve independent deployment doesn't not make any sense to me. If microservices were completely independent of each other, it would mean they are separate applications. But it is not the case, microservices are part of the same application, and they communicate with each other regardless of commination protocol. Events, api calls, and so on need to be unstood by particular microservices. Having an application that is modular and does not use microservices allow separate teams to work on them. There are also ways to deploy them independently. Such architecture is quite common, and this is what plugins are about. What microservices really allow is the ability to scale independently. Properly designed modular monolith allows multiple teams to work on them relatively independently.

[@shakes268]

Microservices also allow for implementing something akin to the single responsibility principal at the service level. Code degrades over time. It just does. The more cooks in the kitchen the chaotic a single codebase can become. If it's a monolith that code rot impacts a much larger code base. People try to implement more abstraction but in essence you still end up with a brittle monolith. One change can (n) other things. I see microservices as a way to have a smaller code base, more focused on a specific responsibility that can scale independently of the rest of the system. Modular monoliths make sense in many scenarios, especially if you have medium sized or smaller teams. It's easier to manage but you still run the risk of something potentially breaking an entire application. It's always a tradeoff.

[@ruijose4803]

I'm not into it, but what happened to your previous channel?

[@gonzalooviedo5435]

Wonderfui Video. Thanks, very useful!

[@dmc_xenon2411]

Thank you for the content.

[@ika_666]

using this video as white noise for my sleep

[@SamWhitlock]

If you have the privilege of having your monolith hold you back, you'll definitely have the privilege of hiring top talent to break it up as needed. If you do uServices from the get-go, you're just leaving a wake of hot garbage resume-oriented architecture around that will hurt you in the short run.

[@jaskij]

You mixed up types of scaling. Scaling up is getting a beefier server. Scaling out is getting multiples. And frankly, nothing is stopping a monolith from scaling out. That's something I never got with microservices. Typically, the limit is how many requests the hottest database is processing. Sure, it can be wrapped in a microservice with an API, but it's still the limiting factor. Having multiple instances of a given microservice doesn't really give you anything here.

[@thekingofallblogs]

Microservices have to be totally independent or they are of no use. If they are not independent, just make that piece a monolith, because you won't gain anything by making them separate. They shouldn't even share databases.

[@CalifornianViking]

Interesting video. While i am a fan of service oriented architectures, I believe they microservices has become blindly over used. First, there is too much focus on the word micro, there is no service that is small enough. Second, as the video points out, there are major latency issues, and while you can scale out of compute capacity, you cannot scale out of latency. Third. There is spiraling complexity. Complexity is added to deal with complexity instead of solving the root cause. Fourth. There is a lack of understanding of microservices overhead. Every developer should understand the time it takes to read from cache, memory, network, disk, etc and evaluate the effects of microservices. Fifth. Many of the claims of microservices are not accurate. Many developers can work on the same code if it is spilt into libraries with good interfaces. No microservices needed.

[@HairyCheese]

I am really enjoying this channel. I've seen failure 3 often. Fat containers taking too long to start up screw you ... The answer, allow more latency LMAO 😂 The funniest answer I was given when I challenged was....wait for it.... you need to consider warm-up 🎉

[@jamesisaacson6414]

I wonder whether some form of request queues (different queues for each priorities) would have helped. What's your take on this Navi?

[@NeetCodeIO]

Possibly, but some of the requests were meant to receive payloads in the response, where as a message queue would just respond with an 'ok' that the request has been queued. Also, if the receiving service is failing a message queue would still be retrying the requests. I guess it would persist them though. Just my thoughts, someone else can chime in.

[@punkboi97k]

@@NeetCodeIO The core microservices at the company I work at just publish and subscribe to our kafka stream using event carried state transfer. So microservice one doesnt need to know anything about microservice 2. But this approach doesnt really solve a "distributed monolith" approach to microservices.

[@mjrmjr4]

Message queues certainly help avoid some of those problems, but like all engineering decisions there are tradeoffs.

[@booznose]

Everything is technical debt. It's simply a choice of where you want to invest that debt.

[@tyswenson7542]

Python can implement static typing

[@theprimecoder4981]

I'm currently learning microservice uusinh Java so this video is reqlly helpful

[@mkres]

Surely an interesting topic, but tbh I have doubts how much experience you really have with different architectures to discuss it. All those problems to a degree apply to a monolith and any solutions in between. Mechanisms like circuit braker come out of box in the frameworks like .net core. Distributed tracing is also pretty standard now. Of course there is a lot to think of and adjust with multiple services so things work well together but you do have more tools to deal with it than with a monolith in most cases.

[@JumpingMike333]

I love this content fr

[@AlanCzajkowski]

if you're designing microservices like the diagram in this video then boy are you doing it horribly wrong, I'm getting tired of people taking a crap on microservices without understanding the architecture or how/why to actually break off a piece of functionality into its own microservice ... just because you're a bad engineer bad at design does not mean a particular architecture is bad if you're considering load shedding, you didn't design the microservice architecture correctly if you're considering circuit breaker, you didn't design the microservice architecture correctly building terribly designed distributed systems is not what microservices is about

[@tuber694]

Excellent video!

[@Unknown373d]

greeat, is this a system design error

[@skanderbegvictor6487]

I had to do these for amazon as well. Damm didn't think all these systems are so similar

[@stephenghool8888]

Good content man. Are you in the Bay Area? Let’s meetup. I’m here until the end of September.

[@east4ming]

It's all about trade-offs.

[@olyneannie4739]

А через траствалет так же можно?

[@Dezdichado1000]

the real issue was they used python, not the specific architecture.

[@shakes268]

In this video it shows they migrated from a python monolith to a java microservice architecture. Are you referring to uber being a python microservice architecture?

[@olge1355]

Great explanations

[@CarlosAlvarado04]

Developers restarting services? You mean operation engineers, right?

[@dafivers4127]

Make more of these videos!

[@kwhandy]

monolithic python? you mean django?

[@Create-The-Imaginable]

The problem is that is was written in Java! 🤣

[@Dom-zy1qy]

Thats actually pretty wild that they were using a Python monolith up til 2020. Sounds like a stressful codebase.

[@MaxJM711]

Fr, I don't mind writing in Python but a single monolith written in it for such a big company? Jfc that sounds awful lol

[@antdok9573]

google has a monolithic code base, too

[@antdok9573]

the point is using a monolith wouldnt inherently cause problems. moreso the engineers that interpret, such as yourself, and the ones that actually contribute to it do. everyone on the same page and it will be easy to navigate it without even having to delve into microservices

[@someguyO2W]

We have too many people who don't have enough production experience 🤦

[@Huntertje13]

If you have a bad monolith then you also have a bad microservice.

[@bitlebron8801]

Pydantic not native though

[@geoffxander7970]

I love postmortems like this.

[@nosouponhead]

It's impossible for microservices to go right.

[@jasonwalton3013]

So sad for people my age to watch this type of video. In the 90s we had HA cluster software did all this for you. The only issue with HA cluster software was it needed intelligent people to utilise it properly. It's been proven to be cheaper to get less intelligent people to build things, which means we need less intelligent approaches to complex problems, which in turn give us systems that are inherently unstable. Every decade I see more and more systems adopt newer tooling which is easier to understand. And these same systems have a lifespan of years as opposed to decades. Perhaps, instead of going round and round in circles and replacing the tooling and methodologies, we should look at replacing the people?