Microservices are Technical Debt

[@NeetCodeIO]

Here's my original DoorDash video: kzbin.info/www/bejne/qIelpY1-bdt1b6c Here are the links Matt mentioned: christophermeiklejohn.com/publications/cmeiklej_phd_s3d_2024.pdf kzbin.info/www/bejne/sJeYeXtnd7WBatE

[@RoflcopterWosh]

thx

[@Winnetou17]

Ah, so refreshing to not see the pinned comment about how we should check [Sponsor A]. After realizing that I've seen already several videos from you, I just subscribed. I also hope that 37 minutes of my watching on Premium helped. Ceers!

[@benb3928]

lol.. typed these by hand before I saw these... would be super helpful to add these links to the description for idiots like me..😅 on topic: found interview is interesting and enlightening

[@EightSixx]

@@benb3928 me too ahhahaa!

[@AndresPineros-k9t]

About the "assertion quality" issue, mutation tests help you with that. They won't tell you the quality of your assertions, they'll just tell you if you didn't consider an assertion, which usually means there's potential for the code to break without the unit tests realizing it.

[@mykal7734]

Neet, this is a good niche if you can pull it off, talking to architects where they share their war stories, its entertaining and educational. You could possibly enter the top ranks of hardcore tech tubers if you did this.

[@yaronlevy]

Yes. Absolutely. Just "real" info from the trenches.

[@whyredvince]

Yeah this is way better than doing reaction videos.

[@crazychase443]

would love more of this!!

[@koderkev42]

He's a relatively new dev so it is on par to make your money on the experience of others. Listen to his response at 01:30 ... he danced all around the question and ultimately ended with name dropping Google (which is no doubt an impressive place to work or have worked). That's a basic question I would expect a real answer to from a guy who literally posted a video critiquing why micro services failed at DoorDash. NeetCode is a very intelligent person ... I would argue in the realm of algorithms, he's Zeus and I'm the guy lacing Zeus' sandals. My comment is not to suggest NeetCode does not understand micro services or that he does not have technical abilities. I am however stating how much the tech industry learns from and idolizes personalities over practical, real world experience. Anyone who has worked in a large organization should know that employees from such places are not always the strongest programmers. Teams and silos are real so while you may work at DoorDash, you could spend your 3-5 year tenure working on some obscure part of some internal facing tool or even on a team who maintains legacy tools/tech. As with much in life, those who are actually "doing the work" are rarely the ones you'll see posting reactions, tutorials and running workshops. The ones you will see doing that are usually those who have not been or cannot be the one "doing the work".

[@yuriib5483]

most companies will make you go thru approvals before you can talk to public

[@chrismeiklejohn]

In my Ph.D. dissertation, I said that microservices have fundamental issues because they are a technical solution to a socio-technical problem. However, during my Ph.D. defense, I made a slip of the tongue (and error in my slides) and said "microservices are a socio-technical problem" and... it sort of caught on. 🤣

[@MiguelOliveira-wz4ju]

That's a pretty interesting take. I think microservices became popular due to the incentives to use could platforms. These days to build a simple website you use at least 3 different services (network gateway, database and computing service) when you can just serve this in a single docker image. These skills are hot.

[@jamesbutwithglasses]

Wow, just browsed through your thesis. It looks super interesting! Going to set aside time to read in full.

[@MrHaggyy]

Really interesting to browse your thesis after the video. I work in automotive where it started with a monolyth, the E(ngine)ControlUnit in the 80th, exploded into a network of dozens of E(lectric)ControlUnits, primarly for real time constraints and secondly to keep things separate that should not fail together. Nowadays it's slowly walking towards a topology of things that fit together nicely from a functional and networking perspective. It's a surprisingly large overlap of the problems and vulnerabilities both industries are facing.

[@AlgorithmAlloy]

Respect for being spot on. As someone with no context on how that mistake phrase caught on, how did it? (out of curiosity)

[@Etcher]

@@AlgorithmAlloy I too would like to know this

[@ashlebeau9802]

Now instead of someone pushing code and breaking things for everyone else, we push code and break things for everyone else but they just don't know it yet

[@vladluteen2299]

💯

[@FredoCorleone]

So true

[@randomstranger7989]

Gem observation bro

[@bartbroekhuizen5617]

Exactly, and commit them in an obscured timeline. Goodluck finding that commited error.

[@l3lackoutsMedia]

The interface is pure free form basically.

[@MichaelZijlstra]

This made me think of Conway's law - the structure of a system reflects the structure of the organization that makes it. They had to split into micro-services because their organization became too large.

[@TylerLemke]

Was just gonna post this. I was thinking the same thing. The follow up is that the communication and social issues are reflected in the deployment issues with microservices as well.

[@FaisalKhan-iw6tw]

When microservices started they said it's inverse of conway's law. We would design our organization based upon our microservices. Like customer team, Payment team Etc Well guess the inverse of conway's law is conway's law

[@MrHaggyy]

Very true in automotive you have several companies providing hardware and software. As they don't want to spoil IP units are quite often combined after who made them. There is also a functional topology like your engine or suspension supplier, but those are almost always the same corporal infrastructure. Like if you need ESP+ABS it's usually a Bosch.

[@gro967]

Or they just saw the many benefits of microservices compared to monoliths and had more than 1 day of time and a capable architect...

[@doyourownresearch7297]

You just made me think of Brannigans Law

[@enkindel]

Start with a monolith. Scale only if it’s needed. Benchmark your application. Don’t create solutions to problems that don’t exist.

[@chillbro2275]

I've heard this recently. And to hear that DoorDash was a monolith also does a lot to give more credence to the advice. But can you build a monolith and benefit from serverless (I mean generally not needing to worry about the server-side matters) cloud platforms? I am genuinely asking, because I haven't seen that yet in my reading, but it would be a fact to consider.

[@Miszkaaaaa]

What if you know that 30% of your codebase needs heavy scaling and the rest doesn't?

[@gamemusicmeltingpot2192]

@@Miszkaaaaa By the time that scaling becomes an issue for you that affects performance, it will be a good problem to have. Most of the time people spend extra resources brainstorming problems and never reach to that level where the scaling would be an issue and abandon the projects before that. It is always better to just get an initial working implementation out to the real user before worrying about scaling problems or code quality or test coverage.

[@iulianbojinca7309]

Not only that, but when the time comes to extract a service the first step is always to ensure you've decoupled it internally as a module. Otherwise you're gonna get yourself a distributed monolith.

[@Miszkaaaaa]

@@gamemusicmeltingpot2192 this is BS when you know what load you will have this is just short sighted approach

[@polvoazul]

Another double edged sword of microservices: Teams own services. So, changing your own service is pretty easy, but changing other people's service's can be very hard (if there isn't an explicit push for a collaborative culture). In a monolith, everyone feels like they softly own parts of the code, but you can change anyplace (say, during a refactoring). Try refactoring some global stuff in a business using microservices... There are two ways: sending MR for every repo yourself (which is not trivial, because you don't even have access to them), and people can and will ignore you. Beg people to do it, which is such a failing strategy because they don't have any incentives to do it. So you probably need some strong company process / culture that enables global changes (I've never seen it in real life).

[@Pharoah2]

Yearly roadmap/architecture reviews are exactly what this is for. What level are you at where you’ve never even heard of these discussions?

[@georgehelyar]

The way we do it at my work is that anyone can make a change to any service but it has to have tests and it has to be reviewed by the owning team. If e.g. someone creates a pull request without tests, you just reject it until there are tests. If they don't understand the consequences of their change you talk to them and discuss a better solution. Most of the time changes are simple enough to be accepted though. Consumer driven contract testing is also useful. If you have to submit a PR/MR for every repo then you probably need to rethink why there are things repeated in every repo e.g. could they be packages or yaml templates etc.

[@Skidmaster180]

I work in a monolith and I don't feel like I own any part of it. even the parts of it I built from scratch have to play ball with the (in this case extremely stupid) architecture someone else made up years ago since a total rewrite isn't in the cards rn.

[@awb19892]

if your team owns the monolith, and it grows and you now grow to 3 teams to own it all, how do you transfer ownership of various parts in a clean way? if you need to make changes with microservices and other teams own that stuff, these are external dependencies you should get buy-in (and coding partnership, if needed) for upfront, not at the last minute. I have seen situations where monoliths are owned jointly by a bunch of different teams... and all MRs get blocked until some integration tests get fixed. Most people sit around waiting, and the second it's finally fixed (which can take weeks), everyone jams their MRs through (after resolving merge conflicts) and the tests break again, and MRs get blocked for another couple weeks. We even set some things up where one specific dev had to approve every single PR (100+ total devs) for a while to try to stabilize things. It was a huge mess. That said, the places I've worked at that used microservices had much better devs and much better design/code/testing quality than places I've worked at that used monoliths, so it's not a perfect comparison.

[@krneki6954]

@@Pharoah2 this is a theoretical solution. doesnt work in practice. or maybe you work in a company that you know roadmap for entire year in advance, and that roadmap does not drastically change. i dont have luxury of working in a company like that

[@matowang]

Love this talk by someone who actually works in the trenches Too much theory talks about microservices out there

[@fatalfailure6240]

pretty much everything about microservices now is just people shitting on microservices

[@msc8382]

Much of what Matt says is also theoretical. I'm a software engineer working with microservices and have experience building (distributed) monoliths. The issues he mentions typically arise when instance replication is missing or when contracts aren't aligned with business value. It may also happen if the delivery systems of messages do not have guarantees. Many developers (not engineers) often confuse technical requirements with functional ones. They're not the same. As a senior engineer with QA experience, I've noticed that this is where quality starts to degrade. From a technical perspective, boundaries need to be clearly defined to prevent unexpected behavior. However, functionally, not everything requires exhaustive testing and constraints. The problem is that functional stories are often interpreted as technical ones, leading to unnecessary restrictions. In my conversations with developers, the focus tends to be on how their technical implementation doesn't effectively address the functional requirements. Ideally, you'd engineer a solution based on functional and non-functional requirements, not the expected technical implementation that facilitates the functionality. But because many devs hyperfocus on technical aspects, they end up imposing limits where none are needed-and often struggle to justify these decisions in a functional context. But they'll write tests to enforce their own imposed limits. Focus on maintaining contracts based on business value, and use ETL services if multiple domains need to interact effectively. That's all. Cheers.

[@beepbop6697]

I work in the trenches, and we utilize microservice architectures, with each microservice running as multiple pod replicas (all Kubernetes hosted -- thousands of processes in each K8 cluster, split over 20 worker nodes). Can you put everything into a few massive monolith processes that each need 128GB of RAM? Sure. Should you? I don't think so. There should be clear delineated separation between logical units. At the same time, don't go extreme and make every single function into its own microservice either.

[@msc8382]

@@beepbop6697 Hi I'm also in the trenches. Cheers. You definitely have seen the consquences of not separating logical units and resources proportionally.

[@mayoo6545]

Microservices are pushed by big tech to sell more aws, azure, whatever. once you're locked on their system can't easily get off. Everything costs 100000 billion dollars expensive and then everyone gets laid off because their products make negative money

[@onaspnet]

His advice was awesome: Dive into your libraries. Understand dependencies or try to avoid them. Value craftsmanship and know what good means contextually. Avoid dogmatic beliefs.

[@TheEVEInspiration]

Dependency reduction and the least number of black boxes are always good things. Which implicitly means micro services are a bad-design pattern.

[@nikolaosraptis7025]

Show me a success story of a product of millions lines of code organized using good craftmanship where over 300 engineering work in the same codebase and don't mess up architecture. You are deluding my friend.

[@nicholasdickrell5118]

@@nikolaosraptis7025I suppose every software product before the advent of microservices was a failure then… Linux, Windows, any Game

[@Alceste_]

@@nikolaosraptis7025 Show me a success story of a product of millions lines of code.

[@Thomas-yv5yw]

@@nikolaosraptis7025 I think you're taking this a bit far. µS not being a good pattern doesn't imply that the only solution is a fat monolith. It's like saying there's no number between 1 and 500. just do miliservices or services instead of µS or megamonolith.

[@rogerdeutsch5883]

Great talk. This video reveals a core truth of IT - There are as many Best Practices, Software Architectures, & Great Ideas as there are software development teams. Each team does what works best for them & then most (many) of them start yelling that "we've found The Way to develop software! You must follow our process. Oh, if we codify this and slap a name on it, we can sell books." Agile, Scrum Agile, Software Patterns, Enterprise Patterns, Clean Coding, KanBan, Six Sigma, ISO 9000, AI, Monolith, Microservices, etc. ad infinitum.

[@headsofhiphop]

Clean coding is generally a good thing every team should do. I've never been happy to read a 300+ line method with 20 nested conditions and bad variable names

[@aqueelaboobacker6840]

I think he mean clean architecture

[@touristguy87]

...utterly and completely wrong. That's not even true for "great ideas". You're not considering the ones that they have thought of but haven't implemented yet. That's why IT and engineering will never hold a candle to "mathematics". It's like comparing paddle sports to tennis and tennis to golf. It's the difference between IT, CS and engineering. IT is all about certs now. There was a time when the only relevant cert was the code that you'd produced. Then people realized that they could make money just by implementing systems based on available code produced by dedicated coders...who were largely mathematicians and physicists with the occasional engineering mixed in, that drove the need to produce the code in the first place. Right there you have three distinct fields of "computing". That's still based on the hardware and software protocols and implementation and even all that depends on cooperation between the various fields, which has condensed into "System Architecture", "Computer Engineering" and "IT". With technical at least one of those a practical equivalent to Computer Science. But you don't have to write a line of code to be a computer scientist, it's all essentially theoretical, "systems theory". Which is pure math. And pure math is pure theory and theory is limited only by the human imagination. This is all an excellent example of people who think they know when they do not know. They barely even have a clue of the issue that they speak of. Just a great example of the thousand monkey example, where if you start off with a message and pass it one by one through a chain of monkeys you end-up with gibberish, at least compared to the original message. Substituting humans for monkeys you might end with something different that still makes *some* sense, maybe even more intelligent than the original message. It all depends on the capabilities and limitations of the people in the message-chain. Anyway at this time we don't look to IT for competent "system engineering" for this very reason. We turn to IT for competent systems-administration. Certainly better than someone without an engineering background, which still can't hold a candle to someone with a mathematics background. It's like asking for someone who runs a Call of Duty multiplayer tournament to write the music and lyrics for the Beatles Abbey Road album, subsequently enjoyed by millions of fans. The implicit assumption in that statement is that a CoD admin CANNOT write good pop music. Maybe they can. But would you bet on it? Would you bet on their ability to even write a patch for a bug in CoD? I wouldn't even bet on their ability to even explain the bug and how the patch fixes the bug. I'm not even sure that I'd bet on their ability to apply the patch successfully. But I'd definitely bet both on the likelihood that that one patch will not fix ALL of the bugs in CoD and I'd also make a side-bet that applying that patch will cause instability in subsequent multiplayer games. Do you know why? Because that happens with Windows all the time! The idea that ONE GUY is going to design and implement a major change in architecture for a working system without subsequent bugs that he hasn't even thought of much less fixed and tested before implementing that change is so small as to effectively make major system failures in the new architecture a certainty. And there are several consequences that have developed around that certainty. One of which is that customers with extreme uptime requirements DO NOT CHANGE THEIR CODE. They keep the original developers on contract to change the code and do significant QA to ensure that it meets their uptime requirements. I seriously doubt that Doordash meets those requirements, or even cares about them. So a few million customers don't get their tacos delivered to their door in a timely manner. And so what? Doordash will still charge those customers a delivery fee anyway. What are they going to do, get dressed and go out and get some food themselves? Seriously. There's stupid and then there's just plain silly. Any credible CIO/CTO will develop, debug, QA and certify their new ideas on a pilot system. They'll deploy it to a few low-reward test sites. Then they'll drive it out, continue to develop, debug, QA and certify. Then MAYBE they'll apply it to their main customer. Once they know where it is likely to break and how to fix it in a timely manner. Even a company as simple as Splunk will do that, not to mention a major national customer-service organization. The companies that don't do that find themselves with a large and long-standing customer-service problem that can ultimately drive all of their clients away. And in todays' market, "if shit don't work" companies can lose market-share and go out of business virtually overnight. Even when they do work reliably they can still find themselves outgunned as time goes by. They need enough market-share to weather such outages. So any SA who gives you a simple, easy-to-understand explanation for a system is leaving a LOT of the details out that make it viable. They might as well just point you to their executive-summary document.

[@kmajetic]

The best dev comment ever, everyone thinks they got the jolly grail if it works for them 😄

[@dallysinghson5569]

I'd hesitate to include something like ISO 9000 but I know for sure you can milk TF out of Six Sigma oh damn.

[@Sanchuniathon384]

Any time a system becomes more complex than the problem it is seeking to solve, then it's time to rethink the system because you'll spend more time on the system than on the problem. That's a good thing when the system can elegantly address problems you encounter. It's a bad thing if you end up focusing more on the tool than the issue at hand.

[@JohnSmith-qy1wm]

As someone on a small team (at a small company) with 400+ microservice that has spent years updating them for a single specific reason, let alone just maintaining them, yes I agree. We're maintaining a "scalable" architecture that we don't even need. In our case, it's never-ending technical debt because we do not have the capacity to knock out all the updates before more updates are needed.

[@scylk]

sounds like hell 😬

[@VV-nw4cz]

You can always take one micro-service, merge its code into another and replace HTTP calls with a function call, remove mappings, remove serialize-deserialize loops.

[@rodriguesd048]

A small team essentially functions like a microservice, but in this environment, you experience all the downsides of micro services without the benefits. What you end up with is a "nano-service" architecture.

[@johnnyblue4799]

@@scylk Sounds like my job...

[@TheEVEInspiration]

And it probably is not even scalable at all, there is overhead involved by being so disconnected into parts. As soon as the services get stressed you will find weak spots you simple cannot fix, because it means changing multiple services or the overhead is to much on the network infrastructure you run it on. And this is the problem with most hype and "best practices". It looks nice on the surface when, and only when, one has not enough experience or knowledge. Most of the nonsense comes from educational institutions as a result that pump it into the still easy to influence brains of students.

[@shashank025]

re: "stepping on each other's toes": I was a SWE at Facebook (I refuse to call it Meta) between 2015 - 2018. Around 2016 IIRC, we increased the release frequency for the PHP/Hack monolith (internally known as "www") to 3X/day. I remember being blown away by this. Mind you, this was most likely the world's most used monolith (it still maybe?), with 10K+ developers often committing multiple times to it daily. A key mechanism that enabled this release velocity was *feature flagging*, which allows decoupling of code deployment from feature enablement. I'd argue that feature flagging is a necessary ingredient for scaling a monolith, and combined with high quality automated testing and telemetry, might even be sufficient.

[@suryadatta1424]

Feature flag is another level of tech debt because those flags need to be removed from code after a point.

[@miloskovacevic2245]

@@suryadatta1424 Man that's on point. For me, this was painful experience.

[@awesomedn]

@@suryadatta1424this is kinda moot point. 1. Always plan time for the post-launch cleanup. It should be a part of the process, and not something you do just out of a good will 2. Implement system of experiment credits. For example, every dev gets N credits initially. To create 1 experiment an engineer has to spend 1 credit. If an engineer ran out of credits - an old experiments has to be cleaned up to get the credit back. I.e. yes it adds some overhead. But it’s not that much of overhead comparing to suddenly breaking a monolith apart into microservices. And cleanups have to be a part of the normal development cycle and not an afterthought

[@eightsprites]

Remove the network calls, keep code in modules. Now you can build a monolith with multiple teams. Network calls should be for scaling. And preferelly comming from the client. Its possible to shard a monolith for scaling. At some points the monolith do need to scale one specific backend part. And network calls are required. So shard that specific module. This is if you want to be multiple teams and still release a monolith.

[@yarbarbar]

This doesn't address the main point of handling rollbacks

[@inxiveneoy]

Yup. Only major issue with monoliths will be major framework upgrades which require major refactoring. You cannot develop new features during this refactoring because the conflicts will be very difficult to merge. But if onion architecture is used this shouldn't be a problem (separate the domain logic from the framework).

[@leeroyjenkins0]

​@@yarbarbardoesn't it? Is rolling back a service that different from reverting a commit? Given that you have comparable levels of coupling between your modules as you would have between your services.

[@monawoka97]

I think the best sign of a senior engineer is they make complex topics sound simple, as opposed to an associate who can make even simple topics sound complicated. Matt is a great engineer. Complex ideas explained in basic no nonsense English. I write C++ for a living. I don't know shit about web dev or microservices or backends. I could follow everything Matt was talking about easily.

[@DominikPlaylists]

Yes. Maybe even too simple. The main point of microservices is absolutely not letting teams move faster,. NeetCode IO actually caught him on that. It is much more common micro services are used for scalability and performance. The large majority of companies map many micro services to one team, only scrappy startups that have way too much finding have separate team for each micro service.

[@monawoka97]

@@DominikPlaylists Scalability yes, performance no. There is no way RPC is more efficient that just calling a library function in the same exe. Not to mention the insane redundancy of every MS fetching the same data because it can't be cached for a single operation. EG pulling account data from a DB. In a monolith you can do that once and just cache it for the rest of the operation. His arguments about deployment workflows were extremely compelling. Having each individual team with their own deployment schedules seems like a HUGE workflow boon, at least at first when you're moving fast.

[@yurisich]

​@@DominikPlaylistsMicroservices primarily exist to fulfill a workplace version of NIMBYism.

[@Xfacta12482]

To be fair, I don't think an engineer's ability to communicate makes them a better or worse engineer. Maybe a tech-lead or manager. Some of the most brilliant people I've ever worked with, whos code was literally keeping the company afloat, were also the worst communicators and socially awkward people I've ever met. It's just a different job and a different role, that call for different strengths. The guy who can lock himself in a room for 36 hours and pump out something incredible is USUALLY not the same people whos sociable, & great at leading a team and teaching. That being said - when they align - you got a talented unicorn.

[@zmirc]

​@Xfacta12482 being social and being able to explain in a simple way a logical problem for a job that requires logic... that's a direct indication of true experience. If you don't understand yet, you're probably still onto that path of growing. It will make more sense down the line, we all grow in our own time

[@IanSherratt]

One 'third way' is the traditional 'monoliths where well defined modules are owned by teams' - effectively this is how much of software that exists works - e.g. everything that makes use of the hundreds of shared libraries on your desktop. Network calls, isolated infrastructure etcadds a whole level of complexity to maintenance and devops...

[@a6b59ghj51d]

The problem is Architectural Extremism (a subset of development dogma) whereby nothing in between a monolith and a microservice is allowed. Something in between, e.g. a "Multiservice" would alleviate issues like dependency conflicts and callgraph clutter that come with a pure microservice-based architecture. Premature optimization is the root of all evil but how many headaches, time and money could have been alleviated during Doordashes microservice transition if they had prematurely optimized. Further, what issues could be addressed by not migrating from one polar extremity to the other by using Multiservices as the default and Microservices only when they are absolutely necessary. Grug says Monolith Bad, Multiservice good and Microservice helpful only when necessary.

[@nsubugakasozi7101]

Actually the multi services thing already exists...its just that the software industry moves back and forth, calling the same thing different names. I remember the actor model originally from elixir/beam was this middle ground. Its not really a separate service but different parts (actors) of the one monolith could scale out at different levels from others. And you could even deploy them on different machines with no code changes (but configuration changes)....anyways docker and containers became mainstream and then microservices became the norm

[@vulcwen]

If premature optimization is the root of all evil, premature abstraction is the soil in which evil grows.

[@rmidifferent8906]

​@@vulcwenPremature abstraction is premature optimization. You are thinking of ways that you might add new features in the future after all

[@ZucchiniSky]

IMO this is the key insight that Matt Ranney seems to be missing. Every architecture paradigm is going to have issues if it is applied incorrectly. Matt is focused on insulting microservices because it's what he's been dealing with lately, but he admits that it would have been impossible for DoorDash to scale as a monolith. Personally, I think most of his complaints about microservices could be solved by having better tooling and by dividing functionality between microservices in a more clever way.

[@nicholastanzillo4229]

I don’t comment often… but this is an incredible video! If you can, I hope you’re able to get more people like this on the channel!

[@NeetCodeIO]

I would love to, glad people enjoy deep technical talks as much as i do

[@scoopy-poop]

@@NeetCodeIOI love these technical videos that you’re putting on this channel. It would be awesome if you did more of these types of videos over leetcode solutions. Maybe put leetcode solutions here and these technical videos on the main? Idk either way if u see this ur videos and website helped me land a big tech internship next year I owe u big time. keep crushing it🤘

[@phoenix-walker]

29:20 There is SOA architecture. Basically before micro services came about, SOA tried to encpasulate business functionality into a single service. There wasn't a stringent rule on not sharing databases. However, there will never be a "clean" and perfect solution for every problem. There will always be tradeoffs. It's up to us as software engineers to figure out which tradeoffs we will have to live with and how long. Then accordingly pivot as our technical capabilities and requirements change.

[@JogoShugh]

This is so honest and refreshing. I experienced these exact problems within "a large European technology giant". It was supposedly mitigated by the famous proxy from Lyft. But, was it really? No, not really. It was still point-to-point calls with a distributed call graph of synchronous calls -- for the most part. There were some more resilient parts utilizing pub/sub techniques. Thanks for speaking out on this!

[@AlfieMakes]

Most people think they need microservices when what they really need is better team cohesion. The way organizations write code often mirrors their internal structures. I’ve seen both sides of this argument, and most fall into the same traps as abstractions-how much is too much, or too little? There’s no universal answer. Your architecture should serve your team’s current needs. I like the idea that microservices can be seen as tech debt, but I'd like to take it a step further: any code deployed to production is slowly becoming obsolete. Like Sisyphus rolling the boulder uphill, we carry the weight of ownership over our code, forever tasked with pushing it forward as it becomes outdated. My 2 cents: - e2e over unit test - Business needs over correctness/perfection - Minimize external dependencies unless they’re core to your technology - Don’t forget the human factor - Always be shipping

[@gro967]

All of your 2 cent points are arguments for microservices and against monoliths, so I agree :D

[@nsubugakasozi7101]

Never e2e over unit tests...I often see this take from like small code bases...any code base with a good amount of code needs and should prefer unit tests for so many reasons. My main one is e2e tests dont tell you much when they fail...you always have to go hunting in log files to figure out which component failed. This is before we talk about how long they take to run especially when you have many components...and how bad quality code gets when e2e tests are the main test strategy.

[@AlfieMakes]

@@nsubugakasozi7101 I think you have showcased why we need to focus on business over correctness. I've seen e2e and tracing be used very effectively to ensure quality of the shipped product. Technical debt is not something the developer owes; it's debt that the organization has, as developers we must advocate against it, but that cannot get in the way of the business. As technical people, we all need to understand that stakeholders don't usually care how things are done as long as they can achieve low customer churn. As for my comment `e2e over unit test` this was in reference when you have no tests, e2e would get you there quickly. Of course, given all of your organizational constraints that might not be enough.

[@gro967]

@@nsubugakasozi7101 e2e above unit tests makes a lot of sense actually. Most unit tests are useless and the real value for the product is not delivered by testing small portions of the applications. The testing should always start with e2e tests and then if needed implement tests with smaller scope like unit. This is the best solution as you implement your most common user flows in e2e tests, which means if they pass the user can mostly use the applications. At that point it just doesn't matter that much if a unit fails a test that is not used by the end user.

[@RaonakDM]

​@@nsubugakasozi7101I get the opposite. Unit tests work well for small projects, but not as well for large projects that have multiple layers. Specifically edge case and one off issues that's often caused by unexpected data/inputs. You can write enough unit tests to cover every single possible case, but that becomes legitimately impractical in terms of project budget.

[@davidparker5530]

This video is awesome. Sadly, I wish we only had to deal with 500 microservices. In Prime Video, there are single calls to aggregator services that wind up making calls to over 1,000 distinct services. Im pretty sure the only reason this doesn't just bring down AWS networking is because of all the caches everywhere that short circuit the call graph lol.

[@experimentalcyborg]

Man it sounds like prime video could cut costs by multiple orders of magnitude just by consolidating pipelines and turning microservices into shared libraries wherever possible. Especially the latter, if you have scalable general purpose compute, you're much better off doing all the compute for one call in one place. (or actually multiple calls in one place assuming you'll be waiting for I/O multiple times per call so you can multithread the fuck out of that bad boy)

[@adambickford8720]

@@experimentalcyborg The overhead of managing that would destroy any possible gains

[@experimentalcyborg]

@@adambickford8720 And why would that be? Your dependency graph would stay the exact same, your interfaces would simply be C-compatible dynamic libraries or WASM binaries instead of HTTP requests. Organizationally everything else can stay the same. And yes almost every programming language can expose their interface as C-ABI compatible or WASM compatible so no you won't have to fire your experts also yes i do understand that means data ownership and sharing between subsystems will have to be considered and that not all microservices can be un-microserviced this way, for example auth.

[@cooltrashgamer]

@@adambickford8720I think teams could still “own” DLLs or SOs. As long as you have a mechanism for distributing and reloading them in your architecture it should work out to maybe be more simple and more performant actually

[@adambickford8720]

@@cooltrashgamer This isn't a theory, its been tried many, many times

[@JoelGordon-m8g]

Many points being made here. First, I've always thought that building things in a monolithic sense saves time and allows for getting things implemented quickly and effectively, but just like a small company versus a big company, the way you run a small company is different than how you run a big company. To me it's the same kind of principle when it comes to scalability. There's more than just how the application is deployed. In a monolithic architecture, when complexity is added, there is a tendency to cater to bad practices and coding standards. That isn't always the case, but it can be. It's also a good argument for why monoliths don't play well, particularly when bringing new members onto the team, because now they have to learn how this monolith was built. That also supports the conclusion that microservices and scalable services just happen to be easier to digest and maintain at scale as well.

[@yesnickcarter]

i had a medical system that processed 20k messages a second. it cost 26k a month in hosting. Boston Consulting Group convinced my manager to migrate to microservices, and that me and my team couldn’t handle it. my boss said he needed 3 people and 4 sprint to rewrite my code as micro services. and it would be cheaper and easier to run. it took 5 people two years to build. my management didn’t let me or anyone on my team help or attend meetings. when it was done, it cost over 800k a month. three cheers to Boston Consulting Group for meeting their goal of making a shit zillion dollars. i quit a few months after the new system went live. i appreciate validation like this. not that i wasn’t validated the whole time. the companies shares have gone down in value by half since i left. it is what it is. a lot of people make a lot of money chasing these fads. or pretending they can get google results without emulating google leadership. maybe i should do that. be a consultant. tell managers they can be just like google if they give me 20 million dollars.

[@skyhappy]

Crazy story, especially a management listen to non-technical people. Company name?

[@lucasteo5015]

Simplicity is always the best, stick with monolith until you actually face performance issues. Complicate things up only when it is absolutely necessary. In terms of OOP only create hierarchy when you absolutely need that inheritance otherwise keep everything as dumb as possible.

[@VV-nw4cz]

Good luck sticking with monolith, when every architect feels obliged to use every single shiny toy on the market, and every developer wants to have the correct words on their resume.

[@lucasteo5015]

What I'm trying to say is to use it wisely, and most of the time wisely means you don't actually need it until you absolutely need it. And not all the shiniest thing the kind of things that you put into production immediately, if all you care is to add fancy terms into resume I would argue that this is not a valid reason to unlimit yourself from adding all possible images you ever wanted. For me I think using bigger container you have some benefits, for example shared types and common code. We have services serving millions with very few container, for my case we have to separate out things into other container mostly due to build time, deloyment convenience and team management but rarely being performance issue. At least that how it is for me, it will be different from problems to problems. Anyhow I'm just speaking from my exp as a junior, I have many strong opinions on things but I also like listening to counter opinions, technical conversations like this allows us to have deeper understanding of the tool that we use, not just blindly following what all the others do.

[@hjimenez26]

@@VV-nw4cz things are changing, people are seeing through the BS

[@TomRaneyMaker]

I enjoyed this talk. In this biz, you get the most points for shipping a product. There is no correct way to do this. We're knowledge workers and yes, there will be tech zealots in the mix, always advocating for the extremes. Unless you work at a FAANG, try to combat infrastructure complexity. It will only bog you down. Layers of additional abstraction seem like a good idea, but in the end you're probably better off understanding "circuit breakers", "load shedding", "exponential backoff" and "rate limiting". Additionally, everything you build MUST be bounded. No unlimited queues. No runaway queries. No unlimited requests allowed to be in-flight. Why this may sound like tech zealotry, it's much better to get a call from a user to get more resources than that page that a critical service is dead due to a bad actor. Cheers!

[@sumitkumarmahto5081]

A really nice point mentioned : there has to something in between, you can’t only have two options : monolith or a microservice. This is probably done by a lot of teams where they carefully break their overall app. They just don’t go say each module needs to be a separate service, rather how close are these two modules to each other and can they perform better together. If there would be tool for this, that would be great.

[@spartanghost_17]

There are inbetween steps dude

[@arturfil]

That would be the same as having a service separate to each other but shares the same db. Like there’s a separation of logic but not of data retrieval. Otherwise you have a monolith if you share functions writhing the same project or services if they have to do a rpc’s for function communication.

[@ahurein1641]

There’s something in between called modulith

[@awakenwithoutcoffee]

@@ahurein1641 thank you for bringing this to my attention. Any tips & resources appreciated ! specifically for B2B SaaS.

[@Reversed82]

at my previous job we implemented something in-between, we used gRPC service definitions to define contracts between modules, then used gRPC in-process transport to communicate between modules in the same process and regular gRPC communication via network to communicate with modules running elsewhere. there are a couple projects that use a similar approach (grafana mimir/tempo/loki IIRC all use a comparable approach where a single binary contains the entire application but it consists of different roles(i.e. modules) that can be run as completely separate instances also, and they communicate via regular old gRPC over network when run separately, though there is some added complexity for the stateful services when they run in distributed mode)

[@nwellinghoff]

The wisdom of the experienced engineer. There is no magic bullet.

[@HollywoodCameraWork]

It's often not mentioned that micro-services create security boundaries. The container itself has identity-based access to the exact resources it needs, and only has the code needed for doing its task. That makes it much harder for an attacker to move laterally once a node is compromised. In a scaled monolith, every node has all credentials, and once an attacker has a foothold in any vulnerability, all the walls come down.

[@Jabberwockybird]

Security is what makes this job no longer fun. It's the Charles Babbage problem at the core. Of course if bad actors give the "wrong" inputs, the system will give the "wrong" (un-authorized) output. The web was desgined by universities sharing documents freely. Having to think of security is antithetical to the web itself. I wonder why we even do E-commerce anymore. I think that server side protection at a "gatekeeper" should be sufficient and everything else is just pedantic. Otherwise we have today where everything is slow and enshitified. Remember, Availability is one of the points of the pyramid.

[@HollywoodCameraWork]

@@Jabberwockybird Still, in a world where you can't promise there isn't a vulnerability in your application, you're kind of breaking best practices by letting any node have access to everything, and doing nothing to prevent lateral movement. One server per function, and minimum necessary access, do a lot to reduce the blast radius if/when you get attacked. It's obviously inconvenient. But the day the attack happens, you'll be happy that an uptime monitor didn't have access to the user table.

[@dominikvonlavante6113]

​@@JabberwockybirdNo. That is not how security works, at all!! Your take is naive and childish. Safety is a multi layered approach, because at any one time any single safety measure is probably breached without you realising. Read up on defense in depth, fault tolerance and so on.

[@Jabberwockybird]

@@dominikvonlavante6113 I wasn't suggesting a security approach. I was saying how security makes the job no longer fun and no longer worth it. I just want to code, I don't want to have to read up on every possible way that some hacker can get around things. I would rather quit and go into a different industry.

[@Matthew-vw6gy]

@@Jabberwockybirdputting people’s livelihoods at risk because not letting people do dumb things isn’t “fun” is a wild take. Please stay away from anything important.

[@andrewmarek]

This is a cool video! I feel like everyone has read that blog (independent of your channel) or a blog similar to it, and having someone come back and reflect on it is really valuable.

[@gggggggggghhhhoost]

Sorry, what is the blog again?

[@o5_25kuhen4ik]

Love people like Matt, I wish all engineers were like him. You feel that he is a really experienced and wise engineer, "street smart" and not book smart. Unfortunately, there so many engineers who just follow those "Bibles" and don't want to hear anything else. Thank you for this video

[@naninano8813]

Wow what a breath of fresh air! agree with most points. def shows what a seasoned architect can be

[@jasonwoods2292]

The comment towards the beginning, about "no one" taking down a system to deploy anymore, shows how insulated folks like this are to how many enterprises run even today. One moral to this story is, understand who's telling you how you should architect a system, and what your constraints actually are, before following their advice.

[@mjrmjr4]

That is a fair callout. All of the systems that I’ve worked on in my career have been internet connected and expected to be online always. But there are plenty of systems that are not like this, and for those you can make different tradeoffs.

[@filipesrubio4015]

Its obvious he is referring to companies like doordash... these comments make me appreciate the fact that none of my friends are software engineers... You will sit here writing a manifest on how oblivious to reality this guy is, only for saying the very logical fact that nobody takes their stuff down in 2024...

[@mjrmjr4]

Hi I’m right here. The comment that I was projecting my own experience to the general case is valid and I acknowledged it. Real people are on the other end of this thing sometimes.

[@AlgorithmAlloy]

@@filipesrubio4015 He’s not being oblivious to the fact that Matt is speaking about companies like Doordash (where 100% uptime is CRITICAL especially for certain parts of the software). On the contrary you two agree, but you, in your pride I suspect, wanted to give him a “that’s obvious”. He’s just implying that micro-services are CERTAINLY not the end-all-be-all and as a developer manager, you need to SERIOUSLY consider if making a part of your software a micro-service will help the developers keep things functioning when needed or if the trade off isn’t worth it. Only consider micro-services for uptime reasons if it’s critical that part of the code can’t be down for more than ~5 seconds. Although that time does depends on your scale among many other things.

[@AlgorithmAlloy]

@@filipesrubio4015 I bet you don’t even know that when you push changes to prod, there is a tiny amount of time where the service is down while it’s switching versions

[@DavidDLee]

The guest ignores some of the great reasons to use Micro Services. For example, you have to external facing services, one with high qps and the other with low qps and a very different resource consumption and error profiles. It is better for monitoring, resource allocation and rollout frequency to separate them into different binaries. Add to that different geographical presence needs, different permissions or access restrictions. Having all these services in a monolith is a bad idea. Google's stack allows a team to move services, as they mature or change, between binaries. It optimizes services on the same binary so they don't go through the network. While it is mature, I don't think it came before anything, and it does force you to micro service even things that should not to some degree, but allows you to compose your binary from multiple service and roll them out all as a unit.

[@michaeldavid6832]

Also, when you hit a bug in a monolith, isolating the bug and all it's dependencies can be a nightmare. At least when a microservice goes down, you can isolate the error to a limited set of services and then drill down into its code. In a monolith, your coders have to be superhuman in their ability to keep things separate. It's not possible to depend on an assorted team to have that kind of discipline at all times. I've met few experienced coders who are that fastidious -- most were German... though they would bog you down with a kickback for incorrect number of characters in a line of asterisks in a changelog while production is screaming for a fix. After some discussions, it was admitted that we need to get the code out and do a quick follow-up MR for any non-urgent touchups. Also, if you're running one binary, you create a single point of failure for the entire system. At least with microservices, a rollback of a chain of services won't take down the system. If you're running a resource-heavy microservice for which you only need 1-to-N connections out to N other lightweight services, you're going to suck down a ton of ram when you wrap all the services into a single binary. But maybe Go handles resource management more gracefully than other languages. Even so, if you're doing some machine learning, Python is still the main language of choice -- which means a different binary which implies a separate service -- and now you're dealing in macroservices... which yields the same problems as microservices. In the end, I agree. Any time someone suggests a single-point-of-failure solution, I have to ask myself, "does this guy know what he just said?" In my career, every time someone thought they could get away with creating a single point of failure, it ended in catastrophe -- exactly on the worst day because everyone is in a rush to get everything done at once. I've only ever worked for one tech company. Usually I'm working in entertainment -- and almost all of those coders have been self-taught -- which means they've been doing trial-and-error learning... and not even learning the basics of the language they're supposed to know. Every last company I've worked for I go into their codebase and search for hard-coded resources. Not a single company has ever had code free of hard-coded resources (URLs, user names, root paths, etc. ). I always warn my supervisors about it and tell them it's going to bite them. They shrug me off. It always bites them within a year or less. I work on the assumption that there will always be a junior coder who will add a ton of technical debt to a project... and sometimes the senior coders do the same stuff -- and they know better. At least with microservices, ownership is clear and when somebody messes up, the auto-shame disincentivizes them to do it again... most of them anyway. Some coders have no shame at all.

[@adambickford8720]

I have found none of this to be true. Monitoring is all down to how you route/partition your traffic. Unless you have specialized hardware needs (GPU, massive memory, compliance) you are almost always better off with a stateless modular monolith.

[@VV-nw4cz]

> very different resource consumption and error profiles. If you deal with something like one of them handles large blob migrations, and another handles small transactions where latency is important - yes, otherwise one fools themselves. The overhead from the friction caused by network calls inside of the system cancels all the gains from this optimization in most of the cases. > It is better for monitoring The only reason you need to monitor it, is because it is a micro-service. If it was a component within the monolith, you would just run a profiler. And no ELK or newrelic are amazing tools, but any profiler is higher gain/time ratio. > Add to that different geographical presence needs Right. Different geographical needs. Like in Asia you don't need 30% of your domains, and in Europe you don't need another 40%, is it like that? Or is it more like you need practically everything everywhere and then you decide on how many instances you have there?! That can be perfectly done with monolith and replication. > Having all these services in a monolith is a bad idea. That is a statement without any substantive argument. > Google's stack allows a team to move services, as they mature or change, between binaries. Google has not figured out how to support their complex architecture, and when you approach them about a problem with their service A, they start routing you between the teams who maintain services B, C and D that are used by service A. And there are not that many googles that exist and need that design, and I doubt this planet needs that many. > but allows you to compose your binary from multiple service and roll them out all as a unit. And you lost that ability, that you have started your comment with. So you end up with a distributed monolith.

[@fahmiirfan6923]

You can just deploy the same monolith, but split the route to different deployment :)) High RPS and Low RPS deployment

[@marcosrolando7987]

@@michaeldavid6832 Pretty much all microservice architectures are actually just distributed monoliths, just like the guest said. So you end up actually creating N single points of failure (one per each service) instead of just having a single one. And while I'm sure that you would not (lol) create a distributed monolith, the fact of the matter is you work with other people, from which you are not aware of 99% of changes that are being made (unless you work at a really small company), the probability of it going from a microservice architecture to a distributed monolith eventually, is 100%.

[@mohitkumar-jv2bx]

This guy is totally practical and i absolutely loved his thoughts.

[@122mlb]

construction worker mentality, i like this guy

[@ianglenn2821]

i think he might be doing the silicon valley noo shuffle, something about how he can't breathe regularly shows he might be micro-dosing

[@mjrmjr4]

@@ianglenn2821 don't you ever get a little nervous when you talk in front of a bunch of strangers?

[@kalirocketdev]

He's a principal engineer so yeah

[@rickardoberg]

I’m working on a library (called Xorcery) with the explicit design goal of “start as monolith, move to microservices when you have to”. The core idea is to always be event driven, but in monolith mode you can choose to immediately project events into a database, and when you go microservices they instead get stored in event store and then get consumed by services that need the events. This should hopefully remove the need for RPC as much as possible, and in most cases each service do zero remote calls (I even run the db in the application process) apart from committing events and async subscribe to events.

[@ceigey-au]

This sounds pretty cool, I've been interested in such a thing myself but haven't had the motivation or time to build something like that. IIRC Spring, Rails or Django have the idea of asynchronous tasks/background jobs that can be configured to run in-process or run using an external event bus or queue, but it's not quite the same as having the framework oriented around asynchronous workflows as the norm. And on the other hand there's frameworks like SST which do make proper distributed event driven systems quite easy to build and deploy, as long as it's on AWS cloud... Feels like there's nothing in between.

[@rickardoberg]

I came to the same conclusion. There is nothing in between. Yet.

[@apestogetherstrong341]

check out polylith

[@postcert]

@@rickardoberg Wasn't using Event Sourcing on that part of the project but when I was in Spring microservice hell my in-between was having embedded/external @Service class variants. Primarily to scale down dev/test deployment size (baseline spring ram costs) but also for things that were expected to grow or take on more load as the product progressed. But focusing event-sourcing cuts out the tedious parts of building the controllers for the external service variants and service discovery (though that was free in my case.) And really cleans up the overall experience 👍 The funniest "solution" to mono/micro services I've seen was where service calls just went to `$local_hostname/$service` and nginx was "service discovery" 😭

[@eNtrozx]

Can you please explain to me why would you assume microservices only talk to each other via events?

[@smanqele]

We don't even do Microservices the way they are supposed to be done, just like OO programming got out of hand with wanting trivial classes that just serve one call over their entire existence

[@xkjjx]

say a group of functions in total processes >5 inputs and outputs >5 outputs - creating a class to manage that, and then getting the outputs by calling getOutput1(), getOutput2() is a cleaner way to do that in my opinion

[@EmiNNsoNify]

@@xkjjx There is no such thing as clean, there is only readable. And while the example with just 5 functions in a single class is still readable once you scale it to a team of 6-10 engineers everyone doing it's own thing, it can easily become a mess if you don't have experience. Most common pitfalls are - every single line is a method and you have to waste a large amount of your time to jump left and right between classes to understand what is happening.

[@TheHermitProcess]

@@xkjjxor, why not write a driver function?

[@jurel-enlatado1]

@@xkjjx how is that cleaner than returning a list, so you can get output[i]? You are basically adding needless interface, also using classes doesn't necessarily make your code OOP. Your proposed solution to that really vague problem isn't really what OOP is about, you are literally proving @smanqele 's point.

[@jaapweel1]

there totally is a way to measure assertion quality, with mutation testing, but it’s… complicated

[@LetTheNewDayBegin]

What are the key takeaways? I'd say: (1) don't do 500 microservices, do 15, (2) rethink your call fanout, because everybody may get paged unnecessarily when one thing goes wrong (and that's no better than a monolith).

[@acupsf]

Yes, do macro-services (instead of monolith, or microservies)

[@IceRockPolar]

Another point is that it's easier for companies to hire contractors to work on a few microservices + APIs then do mocks and contract testing than to give contractors access to monolith and require said contractors to muck with the code.

[@DamianL-o4e]

Essentially, he expresses a lament regarding the nature of his employment. Technical debt, in its essence, signifies the ongoing obligations that persist at the fundamental core of our philosophical understanding. It is an intrinsic aspect of human engineering, reflecting the complexities and imperfections that accompany our technological endeavors.

[@TheEvilKittenLord]

Don't get hung up on microservices -- make services with boundaries that make sense to your use case. The logic can be domain, compliance/security, or just preference driven. Just know that Git, Jira, etc., we're designed to host one product per repo/project. The use against these can inform on whether you've been too aggressive or not aggressive enough in your service boundary definitions. Don't over invest in testing. Test what needs testing to help satisfy your external obligations, but also get you that good quality of life where you're not fixing the same regression repeatedly, esp if it wakes you up in the night.

[@pascalmartin1891]

A more light hearted experience.. My hobby is home automation and I have a hodge podge of equipment and wires across the home. Even as a one-man team, maintaining two monolith became a time black hole. A home remodel forced me to move to a distributed model, so I took a micro service approach. I do keep the number of services relatively low, still about 10 of them, but with only 5 to 6 different different API. Changes are now a lot easier: this solidifies boundaries and makes testing simpler. I make moving services from a Raspberry Pi to another easier and easier, and upgrades are now less labor intensive (the raspberry OS no longer supports in-place upgrades well). So there is value in a micro service approach even at small scale.

[@davetelekom443]

Do you have some repo with the code that you could share? What things are possible to do?

[@icbm7]

In my experience, success is the death of monoliths. If the company succeeds and the code base survives 10+ years, the monolith's code base will devolve into a "big ball of mud" that will incur years of technical debt to recover from. A monolith devolves ("it will never happen to me") because people leave, new people fix something quick-and-dirty, no one who "understands" is there to review the change, or the customer is complaining loudly. This happens a few times and then it begins to snowball. I have lived through 2 examples of old monoliths. In one example the company acquired another company for their codebase with a new/better architecture (followed by layoffs). In the other example, the company rewrote their product from scratch over 4 years.

[@icbm7]

That being said, a lot of what was said here on the problems of microservices resonates. Configuration/user-setting type data and dependencies are a real pain in micro service world. Getting a PagerDuty in the middle of the night and the source is not my team but from somewhere up/downstream is not fun.

[@researchandbuild1751]

Because developer keep job hopping, they are the cause of their own pain

[@Lastninjaxoxoxoxox]

@@researchandbuild1751 companies could just pay what they are worth instead of it being incentivised to job hop

[@everythingisfine9988]

​@@researchandbuild1751nothing painful about increasing salary

[@korbendallasmultipass1524]

Forget all the best practices, rules, religious discussions. Keep always the end in mind ask yourself what would you do if it would be your money which need to be spent. How long does it take to find people and onboard them? What helps you to have a reliable, maintainable, extendable application for YOUR use case? Take over responsibility and DO NOT rely on trends in the market!!! Then you are on a good track.

[@yatsuk]

I guess sometimes people who want to achieve some benefits of microservice simply choose the full Rest implementation because do not know how to get those benefits in another way. Additionally, modern frameworks also push you to implement the Rest microservices. Let's take a look at what we used in the past - Java EE application servers. Using the application servers you could split a project into parts and deploy those separately - just follow Remote and Local interfaces. An application server knows how to discover those beans, does load balancing, and invokes methods. If you use a database per service - the application server manages distributed transactions for you - just mark your method with an annotation that it requires a transaction.

[@pascalmartin1891]

Technical designs are often a balance between conflicting requirements and constraints. I remember someone making a comment about object oriented programming: the complexity moved from being single large modules to a large class graph. I saw references to the same issues about micro services. At my previous company, the product is a very large monolith organized as a set of 15-20 modular processes sharing the same database. It is a monolith because it _has_ to be built and deployed all at once. Because the product is customized for each customer, this has become a maintenance and support noghtmare. Splitting it up in 5 to 10 independently deployable components would have lowered the maintenance cost. The initial cost made it never happen and now most employees are busy doing maintenance and this slows down the evolution of the main product.

[@TimLouw]

Literally all you need is a monorepo with different "modules" each with their own deployment pipelines. Best of both worlds

[@marcusb385]

this

[@youngyu6435]

Joined doordash this week and was at an all hands meeting and saw this guy. I was like wait didn't i see him in a neetcode video just a few days ago LMAO

[@dukedarkwolfe]

leave ASAP. final warning.

[@temolus]

I absolutely agreee ... to a certain point, essential the title is misleading. The problem is not about using microservices, but about CUTTING THEM TOO SMALL. They should not be cutted as small as possible but as big as possible. The key factor for size of a microservicee is NOT technical but driven primarily by organizational factor (teams not stepping ion each others toes) and secondarily of course by business domain. Unfortunately most/unexperienced developers don't think in this categories but in technical or domain categories.

[@KannanH1990]

I hope you do such interviews more often! At Amazon where I work, there are these hour long sessions called "PoA talks" (PoA = Principals of Amazon) where Principal Engineers discuss technical problems and solutions they've worked on. There's so much we can learn by listening to the perspectives and visions of seasoned/veteran engineers.

[@salpula5253]

As an engineer implementing systems rather than developing them, I find the conversation interesting. I have never considered it in this way, because my initial viewpoint of implementing microservices was in separating out the individual components that are basically already communicating via RPC calls using localhost in open source systems we deploy as VMs today. Easy, logical dividing lines in most cases and often times they are already capable of being implemented over the network even if they are not. The benefits this provides of being able to distribute more effectively are pretty straightforward and the network complexity added to the design is worth the overhead, but usually we are talking about breaking these things up into maybe a dozen microservices. Its hard to imagine taking something like the doordash backend and breaking out from a monolith to 500 microservices, until you consider that this was probably decided by a comittee. At some point they have to agree to this, knowing it will be a burden in the future, because otherwise this crucial project that will still be better than the status quo will stall. Of course the cost of the nightmare of "system management overhead" where everyone in a chain gets paiged because a service failed or latency was observed, just to cover the bases, probably never even gets considered at that stage.

[@IvarDaigon]

Sounds like they are discussing a spiderweb microservice architecture where service A calls service B which calls service C and so on.. maybe don't design your network that way and you won't have so much technical debt... If your services just do one thing and talk to one resource then they are far easier to troubleshoot when things go wrong and modify when they need to be changed.. also using the same contract for all services really helps keep things simple and straight forward.. the technical debt is caused by the complexity of the design not the micro-services themselves..

[@ross0za]

Choreography over Orchestration! Agree with you here. Poorly designed software generally results in technical debt regardless of the general architecture. It was stated that the decision to go the microservices route and paraphrasing here .. "just create a service someone can call", was a decision that was made to get things done quickly and they knew or suspected at the time it would introduce debt. People falling over themselves while coding is an indicator to me that boundaries between components of the software were not well defined. That is not a microservices problem. Microservices do force some sort of boundary though, but it may still be a poor one causing coupling and rigidity in the system. As with object orientation, most people missed the nuances of what the progenitors were proposing. OO primarily should have been about message passing. Services calling services used to be termed SOA (Service Oriented Architecture). 'Microservices' were termed differently to try and create focus on service scope as well as a change in the methods of inter service communication. In the early-ish days of 'microservices' (+- 10 years ago) I worked at a company as a consultant that implemented this service to service inter service communication method. Not long into the project (despite multiple error handling solutions that had been employed and having multiple instances of services) there was a cascading failure across the majority of the services caused by a logic failure on one of the services. This took down the entire system for about 15 minutes. It was relatively easy after that to get people to see why message passing (decoupling) was a preferable choice. There is a phrase in Sam Newman's book that really sums this up "Choreography over Orchestration" Microservices are hard to build, and developers should certainly approach them with extreme caution, but if done correctly they do bring the promised benefits. This is not about tooling or design ethos but rather about the underlying conceptual design. What parts of the system have to live together / where are the transactional boundaries. What do we do when this portion of the system isn't working. What is the human / business compensatory action. Just to be clear I am NOT slating Matt. Any developer, with any experience, will be well aware of the pressures that come from "business", and my words are to be taken with that in mind. Poor design is often forced as a result of lack of time. Tech Debt is introduced but business doesn't quite understand what "tech debt" is. Usually to them it is a term that means the devs are taking a shortcut, but they will fix it "quickly" in the future because - look the feature is already working. They don't (want to?) realise it actually means is "We are going to do this sub optimally in 5 days instead of properly in 20 days. Therefore, there is a remaining amount of 15 days left need to pay at some point. Additionally, as with most financial debt's incurred, there is an initiation fee (we have to correct the 5 days work we put in) and compound interest (continuously increasing additional time that is needed to add features / fix bugs on top of the sub optimal design) until that that debt entire debt is repaid.

[@ryanstephen6163]

At 23:50, another good point is raised: If you write a whole bunch of tests for the sake of writing tests, you end preventing people from refactoring the code, leading to more technical debt and bugs. This has been a pain point for me personally, good to hear somebody else say it out loud.

[@johnjallen]

One huge goal of unit tests at least is so you can know where your refactoring broke the existing code. Unit tests show you that.

[@sbergen]

Mutation testing can measure assertion quality! It has its limitations, and can be slow (especially if your tests are slow), but it's really powerful in certain contexts! See e.g. Stryker Mutator (supports JS, C#, Scala).

[@user-wo5dm8ci1g]

Yeah! If your code randomly changes and changes behavior in many different ways, and your tests just keep passing, they clearly aren't very good assertions.

[@havokgames8297]

I came to comment the same thing.

[@funkdefied1]

I love his definition of technical debt-sacrificing future labor for immediate gains. It’s just like financial debt.

[@wy100101]

Microservices are great, but too many people build distributed monoliths. I've worked at both Google and a bunch of places since. The reason why almost everyone fails is because they try and decompose a monolith, and they never do the hard parts, and end up with microservices that share boundaries and that is a disaster.

[@FeedMeLeaks]

This was a really nice take, very rare to hear someone speak on the economics of implementing microservices. Not too many people willing to detail the tradeoffs you're guaranteed to face

[@JoshuaBlackmon]

A/B testing, canaries, etc. are vastly more difficult in a monolith. Feature flags help, but leads to spaghetti code quickly. Breaking it all into smaller modules is essentially what the micro services architecture is. I suppose the difference really just comes down to where your "function call" actually goes: in memory or across network. A monolith is great if you have a very well defined service that doesn't shift much. While micro services are great when you are starting up and the end goal may shift dozens of times or more.

[@omgbaxtergames]

Having worked with both large-scale distributed microservice architectures and traditional monolithic environments, I’ve observed that while each has its pros and cons, microservice architectures tend to introduce more challenges when not implemented correctly. A monolithic architecture that scales horizontally, using tools like load balancers and autoscaling groups, can be more predictable and manageable. On the other hand, microservices, if not properly designed, can become difficult to troubleshoot, overly complex, and harder to manage as they grow.

[@chillbro2275]

Very interesting! I was wondering if/how monoliths can scale horizontally.

[@ReggieMeisler]

Monoliths that are restful seem like they can scale dynamically in a way similar to microservices and give you the ability to easily split parts off if you want, like you could turn on/off controllers through configuration or something so you get the same benefits as micro imo. You should be able to easily split/merge services as needed pragmatically and yes, share dbs and stuff as needed.

[@WallaceBMcClure]

This makes a lot of sense. Some microservices are probably good, but more are not necessarily better. I did some microservices testing and I found that the network becomes a bottleneck. When you are in your own data center, that’s one thing, when you are running in a cloud provider’s network, you are competing for bandwidth with other customers. Debugging in production became an issue. Timing issues came up that couldn’t be repeated in a test environment unless we repeated a 100% exactly copy of production, and no one can afford that. I don’t work at Google scale and never will yet I’ve built monolithic apps that handle users at every scale that I’ve ever had. Until I have a need, I’ll just move along to solve customer problems. I’m dealing with a system right now where the system does need to expose data instead of just calling into this partner’s database.

[@ikbo]

This was wholesome. You should get more guests like this.

[@javedshaik8105]

Apart from the techincal knowledge, we really see how to communicate well with respect when talking to a senior. Which can be applied in an interview as well. Thanks a lot!

[@RobertYeakel]

I feel like, if you absolutely do need microservices, just go from 1 monolith into the minimum number of logically bounded microservices, like if possible, only split in half at most. so then you go from 1 monolith into maybe two smaller monoliths, but split up in a way that actually makes sense. and if you ABSOLUTELY need to go further, then ok split one of those halves once more in half. assuming that you can factor by 2 in a way that is still cohesive.

[@acasualviewer5861]

I would call that "macroservices".. and I agree with you. The "micro" is what gets people into unnecessary trouble.

[@jarradwilder]

Yeah it should only be split in such a way that there's zero coupling. Each service should be it's own thing that can be confidently tested without referring to any other service. If there's any coupling at all then it might be better to join them together, or rethink your architecture completely.

[@benjaminherzig6959]

As many people forget or don't know: Microservices are a technical solution to organizational challenges as Matt describes around 5:35 and it is not a necessary means to an end. It is no evolution or anything. This message goes to all start-ups which start with a microservice architecture. And no one is FANG except FANG - so you are probably not despite what you saw in a nice tech-influencer video or blog post on medium.

[@xalium]

I’ve worked in startups, big tech and everything in between. I agree with every “hot take” this guy brought up. Tons of companies with no legit reason (like explosive growth or hampered developer productivity) enforce building distributed monoliths instead of thinking deeply about who are served and who will service. I’ve used javascript, java, python, scala but none have made the “team” as productive as Go. Less BS magic code, simple interfaces, frictionless tooling. Targeting test coverage rather than identifying failure points (library, integrations, network, etc.) is quite common. neovim, btw.

[@fabiofcferreira]

Quite curious about your experience with the test coverage being used as a performance indicator 🤔 Having worked at a small company (still technically strong), which used microservices, we never even considered using it but maybe that was due to the seniority/maturity of the team

[@xalium]

@@fabiofcferreira yeah, hyper indexing on coverage is copium even though unit testing is great. I have seen valiant efforts to unit test each handler of the entire api suite instead of properly adding assertions to shared code or libraries: all just to maintain that 100% coverage. Waste of time. IMO, general approach should be to add tests for critical code paths that are not likely to be touched but reused a lot. Small teams have enough contextual knowledge of code they regularly contribute to (e.g., api handlers, etc) so they mostly can get away with not writing tests for those things.

[@BosonCollider]

@@xalium Imho, integration tests that verify that interfaces are not broken are much more important than unit tests that test details internal to a team.

[@smthngsmthngsmthngdarkside]

Arch btw

[@yaashbruh]

I think it's missing some points: * Compartmentalization - one feature could fail, don't take down everything * Error handling - sure, everyone in the call chain is getting paged, but that call chain is normally for a subset of functions, vs everyone who works on the monolith It's also possible to create microservices without network calls. There are frameworks like vert.x which basically build a monolith up from individual smaller components (best of both world).

[@trailerhaul8200]

Without network call, your component can only be on one host. And a host hardware and software failure, it will bring everything down?

[@KonstantinShutkin]

@@trailerhaul8200 Based on cool vert.x demos: in vert.x you have an abstract event bus implementation, message can be delivered in the same instance or to a different server instance. if verticle (module) goes down or even entire instance with multiple verticles goes down it is possible to configure multi instance deployment and verticles will be moved to another instance and resume execution from the last unprocessed event

[@crawlingrocket116]

Even with a true monolith you can have multiple instances deployed. Either for horizontal scaling, rolling deployments, or redundancy.

[@VV-nw4cz]

> one feature could fail, don't take down everything RIght! We still can operate when our user profiles are not available. Yes, users cannot login, but card processing and search work. Or the other way around. True W

[@TheEVEInspiration]

@@VV-nw4cz Sounds like a security issue to me.

[@mohsinhaider3830]

"It's not that microservices are bad. Like you got something in exchange for your debt. You borrowed against someone else's productivity in the future."

[@faithfulincro]

So many things resonated with my thinking, but some of them I already heard elsewhere, but I was very glad it was mentioned that tests can have "freezing" effect on non-optimal architecture, because frequently I noticed some piece of design that would need major rewrite, but too big number of tests really make it hard to rewrite/refactor all of related tests beside finding resources to rewrite the production code, so one gives up. Especially because test code is usually of worse quality so harder to rewrite/refactor/understand (just think about amount of copy-paste in test code!).

[@LukaszKubica]

We are an isv for telco industry and our customers explicitly require us to have microservice architecture in their rfps... So it got to a point where their enterprise architects who probably has not developed anything for decades are trying to architect our products. This is a totally crazy situation. It got to the point that microservice is a synonym of a best of breed architecture even though it does not work for some problems at all.

[@mattymattffs]

We have interconnected monoliths. We can upgrade individual monoliths without deploying the rest. No http calls between them, all in the same cluster. Works great. We do have some code paths that are Azure functions for scale reasons. Like a dozen maybe. Those hotpaths really benefit from it. Or they are simply too blocking otherwise. I think if you follow that principle you can really benefit. The majority of your code needs to scale together.

[@Media-Sage]

A microservice is a bounded context that makes onboarding engineers quick. Microservices make **talent** more elastically scalable.

[@Crates-Media]

I'd blather on about things like Nx Cache that can do bare-minimum builds and bare-minimum test runs based on only the code that was changed, but I've learned a long time ago that if someone's already balking at "technical debt" based on the superiority of one code structuring paradigm over another, they are absolutely going to call tests "technical debt" and prefer to have to coordinate every person in the organization in order to do any kind of release of anything whatsoever. Have fun!

[@T1Oracle]

Micro services are an optimization. Optimizations should only be used in response to real world data, not unfounded presumptions. The data that should trigger this change should be when there are clear cutting lines between performance needs, development speed, and resource use, that indicates that splitting up your app can be beneficial.

[@chillbro2275]

An application is monolithic by nature, so I see your point. But the idea of change one thing/area without needing to rebuild the whole app does have its appeal. Also the idea that a service could fall down for some external reason but then be replaced with a new container running that service. That seems really cool to me. I'm thinking about it from a perspective of starting a new application from scratch. But I've not written a single microservice in my life.

[@nizexlizzy]

One of the things we did, and continue to do with error propagation, is we pull a specific error codes that goes into each location. Those error codes travel back up the tree so that anywhere along the way, you can tell what kicked off the train. Works really well for us. We also log those error into a database that can be looked up by anyone to see specifically what happened at each step.

[@ky1emart]

Great video! I agree. In my years of experience, I’ve seen monoliths get split into microservices simply because: 1. The owning team didn’t have the ‘capacity’ to add the features to their monolith, 2. The owning team was resistant to the idea having multiple teams contribute to their monolith. Neither motivation was technical. Just politics, IMO 🤷‍♂️

[@VV-nw4cz]

In summary owning team just ran out of steam due to heavy technical debt. And why did they get a huge backlog of technical debt built up? Because they were unable or not capable of balancing maintenance work and feature delivery. We are all people... and no, micro-services cannot help with that, but they surely help making the system more complex with wider variety of tools used and potentially might help to keep more people employed.

[@Lucanos]

I think a legitimate use of microservices is where you have multiple systems doing work which is separate but also overlaps. Only using monoliths is like having a toolbox which is your "Picture Hanging Toolbox" which has a hammer, and nails, and wire; then having your "Fence Repairing Toolbox" which has another hammer, and more nails. Why not have a "Hammer Toolbox" which you pick up when you are doing either of those tasks?

[@princeani3929]

Absolutely love this kind of content. More of this pls. I would pay for these kind of interviews with actual battle tested devs vs LinkedIn cringe.

[@chillbro2275]

hahaha @ LinkedIn cringe. I would really like this too, instead reading just marketing stuff. And i'd like it in video too, because I'm already reading a lot.

[@bariole]

At the time of SoA (Service oriented architecture), you could write SOAP call as document based or rpc based. Principal difference beeing is that rpc is using api, while doucment style is stating a change. Even then RPC style was deeply dangerous as it was "unmaintainable". The other issue is that RPC style leads to use of sync interfaces like http. Both combined are distributed monolith. Service are used, not only to ease deployment, but to ease development and communictions. If your services are document based, and they are async, the system is stable, and reliable to partial takedowns. The hardest thing about it is to have a lead which knows how to design all those interfaces in manintanamble way whic is also aligned with business. This is principal point of falilure. Business is always talking in a sync way, and good lead should design async state machine.

[@TehFlush]

It's easy to overengineer things. I've worked at a place that did simple payment processing, but ran almost 100 microservices. Tons of issues deploying, problems with timeouts, problems with training new employees, etc.

[@chillbro2275]

How are there so many microservices? i wouldn't think an application needed that many. It's surprising. But I may be thinking of just general services rather than microservices.

[@TehFlush]

@@chillbro2275 honestly I think just the principal engineers trying to make the app look more important

[@PeterNdukwe-u8j]

Matt is not a principal engineer for nothing. I learned A LOT from this video!

[@zb2747]

There are some concepts and methodologies in SWE that are overrated and/or ONLY apply to large/enterprise companies. And microservices is one of them. Less is more man. Some companies and/or engineers want to incorporate things for the sake of 'tech' instead it should be for solving a business problem to add value to the human (users) who actually use the service. Ofc internal workings can always be improved and may change depending on scale. All in all, there are a thousands ways to skin a cat in this space. Keep your mind open and continue to have conversations Nevertheless, great video brother

[@iejelliott]

It's cool to see this kind of content on KZbin. Matt seems like he knows his stuff and would be fun to work with and learn from. The anti-dogmatism and focus on craft was refreshing. Also, this tip was supposed to be 5AUD. Does KZbin take a cut?

[@ChrisAthanas]

One thing never mentioned in these discussions is that the base technology being currently used seems to require these large teams of specialists What if it was possible to do the same work with a lot less people It seems that the each person added causes more complex issues because I’ve noticed an extreme variability of skill and intent for the engineering I’ve worked with

[@fappylp2574]

In my experience less people doing more just puts your business at risk. I know a 1000+ person organisation that relies entirely on the continued full time work of a *retired* solo dev who built a giant ball of mud over 35+ years. The guy gets paid an absurd "consultation" fee for doing this, but if he ever croaks then that company will go down with him.

[@CraigPerry]

I think his case FOR microservices is exactly it (starting 5:11). Spot on.

[@tobia54]

oh look.........a real software engineer on youtube!

[@luiscarlosjayk]

Loved this, definitively sharing it within a Serverless study group in my work. Thanks for it!

[@dream_emulator]

Great interview! Have lived this struggle.

[@Huntabyte]

Great conversation. Call this series "In the Trenches" and interview other individuals not in the YT/social media scene already. Very insightful!

[@JohnWittle]

WHY ARE WE REINVENTING OBJECT ORIENTATION AT EVERY SINGLE LEVEL OF THE CONCEPTUAL STACK i feel like this *exact* conversation could have been had in the 90s around any microsoft watercooler, just s/server/program/g and s/microservice/class/g

[@farble1670]

Because it solves a problem?

[@chrisl4999]

Because new people show up, think they know the solution, have no real understanding of the past, and then … just do it again using slightly different words.

[@mak887]

Everything is a tool; you choose the corresponding tool to solve your problems/requirements. The one aspect missing in this video is how hardware affected the whole microservices idea. Previously, you had servers with a lower count of the cores. You couldn't scale that simple, hence the birth of microservices. Now we have a single CPUs with 128 cores on it. In practice, we have to evolve back to "multithreaded monoliths", but this leads to the socio-technical debt since it's much more complex to do correctly than throw in another scaled-out microservice.

[@rotteneggconcept]

Microservices should be Edgecase, but this somehow becomes a pissing contest, adding Complexity for the Sake of proving Intelligence. Im guessing thats why Shipping Products has become almost impossible

[@TechWaltMD]

The problem with microservices, based on my company's experience, is 4:55 are often used in place of better solutions. They are great, but sometimes, a traditional client-server solution is more maintainable, scalable, and secure.

[@codefast93]

Idk if microservices are “better” overall but they are painful to develop: Setting up all the individual repos as submodules in one parent repo, dealing with all the messaging, individual deployment etc

[@yassinesafraoui]

Yeah, they're not meant for a team of a few devs, they're meant for very large teams as discussed in the video, and it makes sense for those large teams because the overhead you're talking about is affordable for them. But it does make it hard to learn them on your own without being in a company using them as you have to deal with that overhead 😅

[@jrs1986]

I'm actually a proponent of the monorepo with many services. The trade off is you need something like bazel for build and unit testing which has to be maintained but if you have a lot of services could be a net benefit

[@yassinesafraoui]

​@@jrs1986so you mean that building and testing micro services is easier than monoliths. That's also a valid take I think

[@jrs1986]

@@yassinesafraoui no I think building and testing can be harder on microservices because if you have a large dependent graph getting all those spun up for integration testing is harder than a single binary or coping all your interpreted files to a docker container or server and running a single interpreter over them. The monorepo is just a comment on how you organize your microservices. E.g. separate services doesn't have to mean separate repos. Assuming your company has the people or expertise to maintain a build tool like bazel you can have your services in a single repo and run build and test together. This saves some overhead of having to maintain N repos where N is the number of microservices. I've only had bad experiences with submodules and subtrees so I stay away from those like the plague.

[@suheelhussain4975]

@@jrs1986 Sorry, what has mono repo to do with monolith or microservices architecture? Mono repo is just a way to store and build code. Where I work, mono repo contains multiple microservices.

[@samueldominguez9859]

loved this. I have been thinking the same thing since we transitioned a profitable monolith to a microservice architecture that frankly hasn't brought in more business. Please get someone that completely disagrees with this video for your next episode!

[@lukaszkulasek]

The only reason you need microservices is for scalability (and I mean real traffic here - tens of thousands requests per second) and for quick deployments. If you have great infrastructure that takes care of all service discovery, networking etc then it works like a charm. But this is a case for very small number of companies.

[@WikiPeoples]

Awesome discussion! Matt has some really great points and it was a pleasure to get to hear the dialogue between you two. One thing I didn't hear mentioned is how microservices can help with software margins. For instance, if your monolith is hosted in AWS and requires a huge GPU instance, maybe breaking out the GPU requirements into their own microservice would allow them to scale independently of the relatively smaller CPU units. I think this was touched on briefly in the discussion, but the issue of how this impacts cost savings wasn't really highlighted...

[@MrRmeadows]

In my experience. In a large team, I am not seeing things change a lot. I am seeing months go by waiting for bug fix.

[@dellendinho]

Why am I watching this conversation with a big smile on my face. Finally, a pragmatic conversation.

[@s3ed1y]

If your feature requires changes to 6 microservices, then your service boundaries are simply wrong.

[@wsidechris]

So what, you’re not locked in. Just shuffle endpoints around if it makes sense. It’s an iterative process.

[@mjrmjr4]

The problem is that MANY changes can be made independently, and we don't talk about those because everybody's happy with the outcome. I'm just talking about SOME changes that require touching a lot of services, and it's really hard to get an ideal API design and service separation for all possible future changes.

[@lmao6]

At my company we have millions of messages transmitted between 2 services, would have made much more sense just combining them into 1 service.

[@jarradwilder]

Or the feature is wrong. I've seen a lot of features that should never have been a thing and denied by engineering

[@dv_xl]

I have experience working on something that is microservices but isnt (all) a distributed monolith. In my experience, even if you have really good failure modes with queues & coalesesing, the modular nature of independently moving development will always cause problems. Even if service A can degrade and B can survive and serve, A will need to come up eventually. But for me, that isnt even describing the cause of the technical debt. Allowing independent action doesnt actually make things independent, it just serves as an illusion. Untested assumptions enter the system constantly and never leave. Side effects and quirks and bugs are relied upon, and it gets very difficult to understand when everything is distributed and moving quickly