MircroServices (and a story about Netflix) | Prime React

[@KRAZAM]

all happy developers are alike... every unhappy developer is unhappy in their own way

[@GeorgeTsiros]

pppppppft you speak as if you know... you weren't there man... _you weren't there!!!_

[@strigon]

The “Anna Karenina” dev principle

[@herzog0]

This is the best crossover episode I could've witnessed ❤

[@godowskygodowsky1155]

What is this thing you call a happy developer?

[@markm1514]

​@@godowskygodowsky1155 A retired developer

[@asimov647]

So much work for something where everyone I know was like "Oh weird it starts with season 4, better switch to S1E1 manually"

[@akhil1996kc]

Hard agree, everyone I know thought it was a bug and manually changed to S01E01

[@4idenn]

This so much, wasted effort imho

[@ernesto906]

This is a text book example of something "cool" that someone in management, marketing, UX, etc comes with That makes engineering a suffers for a while for little or nothing.

[@engineerhealthyself]

In these situations I always remind myself "errybody gettin paid"

[@PoorlyMadeSweater]

loool, is that what they ended on? Just pointing you to a different season? I assumed they were switching the metadata to make S1E1 a different video, like they did with Love Death and Robots

[@thefrub]

The fact that he can just rebuild this whole system from memory shows how much he had to deal with all this every day

[@SPYTHandle]

LoL I initially thought you meant the K video and not actual primagen😂

[@brssnkl]

When you have 3600 hammers connected with unique cog sizes in a 4D map with multiple space time continuum flaws, everything starts to look like a nail to you.

[@happilyconfuseddog8951]

The fact that this problem arised serving up Black Mirror contains irony that is not lost on me.

[@gregoryshields4258]

“arose”

[@publicalias8172]

@@gregoryshields4258 "penis"

[@GordonChil]

“arisen”

[@yojou3695]

"aroseth"

[@dmsalomon]

"arizethed"

[@vanweapon]

Wow I remember trying to introduce someone to Black Mirror after S4 but i had already watched it, and i remember how S4 would play first and it was really confusing. Can't believe that was you lmao

[@ThePrimeTimeagen]

Gotem

[@woojify]

@@ThePrimeTimeagen galactus at it again, predicted time and space for the episodes that must be watched before the viewer knows what they want to watch

[@nateedwards1313]

Watching you draw S's like a psycho feels like Black Mirror

[@bdlowery]

That's how left handed people write S's ;)

[@banditosdoritos]

What's wrong with the way he writes the 'S'?

[@schism15]

​@@bdlowery Nah, I'm left handed and I don't write my "s"es like that. Don't put that on just us 😂

[@sudonick2161]

@@bdlowery he's not left handed tho :)

[@jorgeimo]

holy shit, now I can't unsee it, the way he writes the four is also weird for me

[@B20C0]

4 months and multiple services for something most people considered to be a bug. I now look at my boring job in a whole different light, thanks for that. 😀

[@Extys]

Yes, but you don't make tens of thousands of dollars a month like in big tech lmao

[@ISKLEMMI]

This skit raised my blood pressure, but the PM's reaction at the end brought it back down. Then your diagram raised it again.

[@LordFokas]

Yes but the middle name thing is like "we'll be going through this again very soon"

[@linusdannull1247]

Somewhere around 11 mins, I just started laughing as it was so clear he was completely re-enacting the video

[@ThePrimeTimeagen]

its... literally the netflix architecture when i did the change... its not a joke, which hurts.

[@linusdannull1247]

@@ThePrimeTimeagen I'm about to start a Mastercard internship using Java Springboot... so I'm looking forward to understanding the pain :D

[@sortsvane]

​@@linusdannull1247I'm one month into my internship at AMEX and the pain is growing every single day.😞

[@martendittmann5431]

I was showing my younger brother this video and before you got around to your story. Black Mirror was mentioned and he said 'Oh, the show where they screwed up the order of the seasons.'

[@earthling_parth]

Lol, my friends and I just switched to S1E1 manually because it started from S4 🤣😂🤣 I feel bad for you Prime 😅

[@rapzid3536]

OMG, I was like "Man, what does Black Mirror specifically have to do with microservices?". Hold onto your monoliths tight peeps. As long as you can, never let go.

[@datguy4104]

People got paid like $300k a year to come up with this solution... and it took 4 months...

[@temper8281]

Yep. Starting to think big tech is just a giant ponzi scheme

[@TheMsksk]

And all the end users thought it was a bug and manually changed it to S01E01😂 Brilliant.

[@fulconandroadcone9488]

@@TheMsksk Must be one of those more expensive bugs

[@ricardoamendoeira3800]

@@TheMsksk It was a bug, it just happened to be written by a product manager at the spec level, instead of by a programmer at the code level.

[@TheMsksk]

@@ricardoamendoeira3800 I like this way of looking at it

[@BhargavSushant]

Irony is that the way he tells it, it feels like those 4 months were a black mirror episode in itself

[@Bluesourboy]

I'm currently working at a place that does a hybrid microservice /monolith style of architecture. The decision to structure it this way came about from specific needs (think content creation vs content moderation) and many code refractors (services being deprecated quickly so it's hard to rely on one). The monolith single responsibility app consistently performs better than the microservices based apps. I think we'll see the industry start to realize sometimes it's better to have a tiny monolith single org responsibility style repo vs microservices hell.

[@fulconandroadcone9488]

I right now work on a fe that support multiple themes ( think of same app for different country or slightly different ui ). Project manager who is also main be dev made a decision to split one int o a separate repo ( which dev ops need to set up and they don't seem to have enough time as it is ). I could not figure it out how it will make anything easier for us since you don't need to share code if it is the same repo what difference is there if a directory is in the same repo or not? Then I got a little story how they have 20 repos on the back end and ever since then I don't bother to even try sway anything anymore. Few months back we got 2 user tokens so 1 user could change and see data of the other user. But you have to send the token from the correct user to see what is going on.

[@alpacamax3404]

If I remember correctly, Netflix used to have a monolithic architecture. They eventually switched to a microservice architecture because they realized the cost of developing and maintaining a microservice architecture is lower than the loss of revenue caused by more frequent crashes from a monolithic architecture.

[@wilkyarny3012]

Learned a lot today love Content Metadata Service uh, but we'll talk next week about adding a new reverse button player. Awesome sounds good, take care. 😂😂👌💪

[@Slashx92]

Lmao 💀

[@EricLS]

*Roto-levitates out of room*

[@ledjon]

I love the Netflix story here. Perfect example of microservices in practice.

[@apidas]

it's a mess. microservice given time doesn't really makes sense for the use case it's intended for. usage and complexity grows.

[@georgytioro]

Nope, MS should never be overcomplicated, they should be simple individual jobs of.execution. this.type.of overcomplification is just showing "rush.to.complite the job", not ever take a second to think and design proper solution. Basically is.showing difference between Developer and Engineer. Prime is a developer, very good.and experienced developer, but Engineer would solve this quite differently

[@simonabunker]

Are you sure the service you couldn't remember why called Galactus?

[@mvargasmoran]

Prime... I would've just hardcoded the Black Mirror ID and sort that out hardcoded on the front-end. Yes, do I that crap when they want things for yesterday, I do it in a 1~2 days and then I would take the 4~6 months to do it properly and remove the Hardcoded stuff, I call this: "Tech Debt Quantitative Easing" as Manager most likely would never want me to pay that debt.

[@TheBswan]

I hate that I can relate to this. We have feature flags, but there's no backend bandwidth so let's just hardcode org IDs in the UI application and we'll clean it up never.

[@dealloc]

@@TheBswan Yes the universal word for "later" and "another point in time" in tech is "never".

[@rand0mtv660]

@@dealloc make sure to put a TODO comment in code so that you can read it in 5 years and see how useful it was.

[@earthling_parth]

@@rand0mtv660 LMAO. Those TODO comments are like troll posts from the distant past all around in codebase 🤣😂

[@dealloc]

@@rand0mtv660 "FIXME: This is a hack that should be fixed later"

[@CaptainToadUK]

I always wondered why the episode numbering for Black Mirror was broken. Figured someone dropped a bollock in the database, never occurred to me it might be deliberate... 😆

[@tanthole0306]

Some years ago, I watched that video and did not feel much because I was a student. Now as someone who has recently done a new feature with many microservices involved, I totally feel the pain. Sometimes I wonder, are microservices actually a better way than monoliths to do things.

[@hotrodhunk7389]

Spoiler: Fucked up how everybody abandoned him. He took one for the team and they just left him to hang...

[@moneymaker7307]

The problem with software engineering is that there is no such thing as a senior or we got to point where because there are so few senior software engineers that the blind started leading the blind. Back in the day, we were building this beautiful machine that is sending us bunch of data from space. Me being a young engineer that is full of ideas brought up the idea of using micro services to process the data and it sounds like this micro service idea help a lot with scalability. A engineer on the team that having been coding since the 80s. Went home and did a bunch of research on micro services. What he found out was shocking. It turns out that micro service are services with the world micro attach to it. After studying SOAs in depth. I came to the same conclusion also..

[@VivekYadav-ds8oz]

So much energy and network bandwidth used for something so simple. Feels like rather than writing code for business logic, we're calling services and composing their results to build the logic up. Then again, I'm just a student and have no idea about the kinds of compromises that happen in production code.

[@reralt]

I am also a student but I guess this much complexity is because of Scalability. Simple things are complex to do but large scale stuff become easier I guess.

[@VivekYadav-ds8oz]

@@reralt I feel this is because they're using the same components that are used to solve bigger issues, to solve small issues. It seems like you send a condition to a service that evaluates it to true and false returns the result to you - kinda thing.

[@TheMsksk]

@@VivekYadav-ds8oz I agree. Basically all they do is build abstractions based on some assumptions that are invalidated by a business requirement later on. So they have to do all of this patching and jumping around just so that their existing abstractions can be reused. Modifying your existing abstractions is a no-go because it can break more things than you can imagine. I find it funny to be honest.

[@VivekYadav-ds8oz]

@@TheMsksk Yeah this! You put it into words better than I could. It is funny tbh yeah XD

[@apidas]

back in the days, when you're not a "student" microservice isn't a thing. and suddenly gets trendy and becomes the "norm" as with any innovation that comes in the industry. and then goes to your curriculum in school. until we learn something in the industry.

[@TheRicherthanyouguy]

I have so many feelings from this because currently this is my life at work. I nearly wrote a giant paragraph about how this made me feel then needed to delete it sigh.

[@EricLS]

Oh man. I’ve been on the internet long enough I basically delete anything past a certain number of lines. All but guaranteed to be a toxic vent on unsuspecting strangers if you go on that long, and you feel like shit afterward. I feel you.

[@erickmoya1401]

Leseon learned: Dont use "smart" names for your services. Racoon, bingo, etc. It gets easier to communicate later to other devs if every service has a boring name related to what it is supposed to do.

[@zyzlol]

Bonus points if you can avoid acronyms

[@SemiMono]

@@zyzlol Yep, exactly. I've worked with both. I find the "smart" names, or random names, easier. Cause then you're saying real words rather than just spewing letters that are ultimately meaningless. The real thing to avoid is making services with illogical responsibilities. If something is a bit of logic, put it in a library. If something stores some data, make it a service. And if you're going to make a service, THINK about it. But in large organizations people are looking for something to do to prove themselves, because the organization over-hired. Then everyone wants to make a service, and you get nightmare fuel.

[@Krasshirsch]

I disagree, every service must be named according to a matching Warhammer 40k unit, obviously.

[@vecter]

Except for Video Metadata Service... which didn't store any video metadata.

[@Ehal256]

I thought the lesson was don't fall for the microservice scam

[@johnrodger5467]

That guy could tell you some stories though. I heard he was later teaching Javascript to a small pack of primates.

[@frootflie2968]

For what its worth, I went back and rewatched the PF video after this. Positive side effect to microservice architecture.

[@progfix]

Was the craziness of all this explained to the person(s) who wanted that feature? Maybe after seeing the cost of developing all this would have made them change their mind

[@max3446]

The original seasons were better than the Netflix ones as well anyway lmao.

[@Yous0147]

I've tried explaining to the affected that the solution they were asking for, even if it seems simple, would complicate our current architecture because it was essentially a departure from the initial spec of the project, but it's hard to make someone who only thinks in features instead of maintainability actually see any value in forgoing a feature for a healthier stack.

[@doc8527]

I fainted 3 times while watching the Netflix microservices graph grows.

[@isaac.omondi]

Just how much data is replicated across all these services?

[@sohn7767]

Yes

[@airman122469]

Lol and I thought the system I’m working on right now is a hot mess. Thank you for making me feel a little better about my job.

[@ZantierTasa]

Reverse season order is such garbage. When it first came out, it was easy to accidentally start watching from a later season rather than S01E01. And it's such horrible UX to scroll through the episodes and them not be in order. You want to promote a certain season? Sure, fine. But when I'm viewing the full episode list, show them in gogdarn order!

[@ToPFeE1234]

It relatable to me as well. Business guys don't care what you working on they just want a result.

[@richie7425]

recently finished helping modernise a monolith. Treasure your microservice architecture. you think that takes a while to get stuff done. 25 years of code piled on top. took me months to migrate just the db out

[@Rockyzach88]

I knew there was something fucked up when I went to talk about the episodes with friends and I realized the order of the episodes everyone watched was different from mine.

[@luissolanodev]

Don't come into this chat with your VS code vibes😂😂

[@boot-strapper]

I bet Netflix performance and ease of development would be improved with monolith. Lol

[@brandonjoaocastillo7490]

This is your best video by far, loved it.

[@Tony-dp1rl]

It is funny to me looking back - micro-services done well are identical to what we already had established and working in the 1990's (even 80's) with SOA. :)

[@ExpertOfNil]

It is so infuriating starting a new show, only to realize I was starting somewhere in the middle 😕.

[@halim7725]

Man I must be extremely dumb because I came up with the same question as in the end and I thought wait I need to re-watch, and now I feel like the interior of my skull is a gigantic ocean of nothing because I'm still wondering why you needed all those microservices to reverse a fucking list of videos

[@ceriusgeek2749]

This is so incredibly accurate.

[@wtfusernamecrap]

I LOLd at VMS. Maybe I'm slowly getting the microservice scepticism. I'm definitely doubling down on 1.) a minimum number of layers in an architecture and 2.) well defined layers and terminology.

[@CaptTerrific]

12:25 - what I'm hearing is that it should be far easier to hack my way into watching temporarily blocked episodes/videos than I expected!

[@someguy9440]

if y'all only knew how the product page worked on your last amzn purchase.

[@CaptainWumbo]

it was both cathartic and gave me extreme existential dread.

[@Skorps1811]

LMAO i now remember s4 thing, and I remember also how somewhat satisfied I felt “you’ve a bug there Netflix, gotcha!” 😂

[@blackfrog1534]

As the team leader for the BE team on the project I'm working on, Primes story made feel better on two fronts. First of all I feel better about our microservice architecture after seeing that diagram that prime drew. But the best part is when he said how the front end guys complained about the naming of the flag... like yesss thanks for sharing this, now i feel like this is something that is just happening to me! P.S. Love you Prime but the way you write the letter s should be illegal xD

[@hfspace]

well, i feel that the architecture could be better. Microservices are fine but what do you think about just using a distributed database like yugabytedb where each microservice can read the data it needs. I mean the goal of these services is to provide data, isn't it. So why not let the database handle the scalability issues when you want to provide data to all the users. Even if you call other microservices to access data in the end you will create a request to a database, you added just a lot of network requests

[@blackfrog1534]

@@hfspace That sounds super cool to me, i would be down to try something like that. But sometimes you are constrained with the tech you can use. For example if because of budget constraints you can only use the cheap AWS serverless services like lambdas with DynamoDB and do as best as you can with that. But what you said sounds super interesting to me :)

[@hfspace]

@@blackfrog1534 well, at my client, we also don't do it like that. I would also like to try it out, but i did not have the opportunity, yet. In my mental model micro services are the ones handling the application logic, but the data access would be managed by such a distributed database, which makes sense imo as you want to protect data at the source.

[@dealloc]

@@hfspace What happens if the data is one thing when one microservice reads it but changed before another microservice reads it, etc? Then they have different data from the same place. How is this handled? Who knows what is the correct data and sync it?

[@hfspace]

@@dealloc one thing: you define responsibilities. Only the microservice who owns it, writes to the table. That would at least be my suggestion. The other microservices can only read which will be needed for aggregation of data.And you can set up roles for that in the database. For reading before writing, i think that problem can happen all the time even in the architecture as described here in the video

[@TimoWelde]

Oh wow, thanks for the background. The opposite season sorting on black mirror always just looked like a bug 😅

[@colinstu]

Prime explaining all these netflix microservices and I'm thinking: why is this not just all not just in postgresql? with the right schema... order by would do exactly the trick... JFC why do companies complicate-to-profit.

[@dragonridertechnologies]

I was not expecting this to pivot into a real-life version. I feel for you. I know these exist, but I've been too low-level so far to have met one yet.

[@Agent_Six_]

He pauses so often he misses half of the jokes. "See bingo knows ev..." *Pause then 5 minutes of talking * *Unpause* "...ryones name-o"

[@darkarie]

this one is awesome

[@memebro3181]

Truly microservices hell

[@grim.reaper]

🤯🤯🤯🤯 I think this is true with any service that has been around long enough and has accumulated a lot of dependency…

[@_bsmusic_]

This was the closest reenactment of that It's Always Sunny In Philadelphia meme

[@Aimsport-video]

Pure genius to migrate the “Ooops, something went wrong.” microservice to the Netflix “Whoops, something went wrong.” microservice. On par with Under Construction animated GIF.

[@chiefsiege]

all i could think during that diagram draw was "JOHN FUCKING MADDEN"

[@tuftman6092]

I remember when I first got into black mirror it started playing s4e1 and I was like wait, why is it playing from the start of season 4, and I remember wondering why they did that. It's cool that I now know why that was. lol

[@carltonnnn]

audibly saying "oh nooo" at every new service in the netflix story lol

[@DryBones111]

Do I even want to ask what changes needed to be made to support the varied episode ordering of Kaleidoscope?

[@tkg__]

I still don't get why you couldn't get all seasons as a list of lists, and then move the fourth list (a fourth season) to the beginning of the list without reordering anything else. Feature flag it and make the front-end reorder them.

[@lettucemode8643]

My theory is they wanted it to be done all in the backend because then it would work across every device Netflix is on without having to ship new versions for all of them.

[@Brandon-hd4vg]

Ngl got an ad for jimmy johns and thought it was a micro service. Good job you got me.

[@robert.durica]

"ah, humor based on my pain. ah hahaha"- ross geller

[@ascourter]

Sounds like a distributed monolith lol. Twice the headaches and none of the benefits. (from the skit)

[@timndichu]

This is crazzy. BECAUSE OF ONE SHOW

[@EricLS]

13:54 “I told you, it’s the design of our back end!”

[@ricardomonge2769]

So much work and effort, for so much nonsense. The definition of hell on earth

[@nateedwards1313]

God struck down the Tower of Babel

[@borjinator]

Banger

[@mvargasmoran]

And that's why we are so slow, Omega Star don't support ISO timestamps like they said they will A MONTH AGO! We are constantly blocked! My story is the same but with old IBM ecommerce software.

[@danielwalley6554]

This is where companies need to be willing to bite the bullet and refactor. The initial architecture is always going to be a mess because you don't know what you don't know. But the lessons learned from it allow you to create a much better successor.

[@rado147]

my brain went for a smoke and I found it hiding under the chair outside

[@SpeedfreakUK]

jfc webdev is even more of a hellscape than I could have ever imagined. And I thought gamedev was bad.

[@MegaMurcelago]

when i watched about neflix services - i wondered, why there are so many of them. Now it kinda makes sense

[@JoakimKanon]

Wow, S4. That's really putting the user first.

[@valeenoi2284]

I shared this with junior level developers a year ago... nobody felt my pain. 😭

[@iamloonix]

we can all feel the pain in your laugh at 8:59

[@laughingvampire7555]

it sounds like you needed to hire Tom, because he is a genius and he could mod jDSL to solve your problems

[@CalebStephensPlaysPiano]

I’m just now watching this… PF is legendary 😂

[@jagagemo8141]

11 seconds in this guy gets in a somber mood, lights a cigarette, "Let me tell you something about war."

[@paulsanchez5030]

this engineering failure vulnerability here is giving me signlas prime

[@Ronoaldo]

OMG that was wild. A real black mirror episode in real life while handling black mirror episode ordering... 😮

[@Visionary1002]

What's wild is I remember noticing this and thinking... why isn't PF 1st, I don't wanna watch this out of order.. but um... mad respect tho.

[@alex6337]

You can tear my monolith from my cold, dead hands

[@buraky16]

I am a professional developer, and i have been having a break from development. I had been losing my passion for my job. After 6 months break, i finally started to feel excited for software engineering. Well, till i saw this video.. now i can feel that my excitement faded a little. This video reminded me why i lost my passion in the first place.. every developer here wants to be a good engineer, and that's what makes us want to be developers, but sooner or later company's profit demands us to engineer badly..

[@etgaming6063]

I just watched Black Mirror for the first time before coming here and wondered why this was. I feel like Im in an episode now.

[@lethanhan9084]

krazam's content is gold

[@gr1nch3]

lol, I now understand why i tried watching black mirror and it started from S06E01

[@konfcyus4865]

I still dont understand why they couldnt order the list on FE for when it is necessary , yes it is messy but so is the pipeline that is being shown for it.Would appreciate if anyone can help me understand

[@israelafangideh]

Same

[@lvlinty]

Probably something along the lines of: Netflix is supported on a wide range of devices, some of which may be eol and not receiving updates. Solid chance trying to do this on the front end would result in patches for hundreds or thousands of different types of devices.

[@DF-ss5ep]

​@@lvlinty No, but they did something with the hidden flag names, and even had to switch them, so they had to do updates. Maybe the FE team didn't want to bloat the bundle size. IMO, the FE doesn't look good in this picture, maybe it's their fault. The issue with the flag names hints at an obsession with procedure. "This is not the right way" kind of mentality.

[@dealloc]

@@DF-ss5ep Doesn't make sense, since most of the lolomo things are server-side rendered. Only things like interaction, streaming and loading additional content after first load is fetched on client. I'm sure they have a sort of BFF that could do the mapping.

[@apidas]

the idealistic argument is "it's should be done in backend"

[@thirdvect0r]

vscode vibes holy shit I'm crying

[@JustSomeAussie1]

Took me a while to realise prime was saying "black mirror". Sounded like "black meer"

[@jebwatson]

The story made the pain so real

[@ResonantFrequency]

The pain will always be present. Sure we'd love to have simple straightforward services that do exactly what we think they should, but we also want performant systems that will last a long time as it turns out product doesn't want the user to get hit with a 5 second query at request time so we build caches and indexes to solve that problem. Turns out you from 5 years ago did not possess a third eye that can see into the future and anticipate what product wants today so you don't have direct access to the data you want that is guaranteed to be up to date and valid for a transaction and now you have to make elaborate changes to accommodate that feature. Ultimately this is why any sufficiently long lived codebase starts to look like a hoarder's house as you build seemingly redundant features that meet varying use cases.

[@moonoovie]

I would LOVE to hear about what went into the Arrested Development Se4 episode re-ordering debacle!!

[@griffitaj]

I remember watching black mirror and being very confused that the first episode wasnt S01E01

[@lishhhhmm]

Did i just watch a video and then a recreation of that video in the same video? xd

[@abdelazizlaissaoui9079]

Imagine doing all that work to deliver an episode with the title of "bf"

[@manny7662]

all that to leave (he/him)

[@Daniel-ic7bz]

The Dr. Disrespect of coding. Loving your content.

[@cgme9535]

He constantly reminds me of Dr. Disrespect. The looks like him, acts like him, and has the attitude of him haha

[@jesse2667]

I was feeling every second of that video skit. It all was too relatable. About The PF episode, I watched the show from the first season. I love the show and listened to the explaination, but it sounds like several services and rewiring with a v2 for what was a single use case. If it is broadly used for more purposes now, it is fine, but if it is just for PF, I'd say some hardcoding and to put to bed those PFs nightmares and never forbid speaking of it again 🙊.

[@cgme9535]

The fact that this somehow works is truly astonishing