Serverless Was A Mistake | Prime Reacts

Recorded live on twitch, GET IN
/ theprimeagen
Original • Serverless was a big m...
YOU KNOW WHO THIS MAN IS!! / @fireship
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord
Have something for me to read or react to?: / theprimeagenreact

[@iamtheV0RTEX]

"Self hosted vs serverless" and "monolith vs microservices" are independent axes. If you put your entire application logic into a single container that you run using App Runner, that's a serverless monolith. If you build your app as a fleet of microservice containers but you run them on your own hardware, that's self-hosted microservices.

[@tejeshreddy6252]

Just goes to show how much misunderstanding/confusion is out there in the backend among even experienced devs

[@yuriib5483]

And there are cost calculations in which that apprunner bullshit is more expensive that IaaS. architects need to be really kicked in their balls for not considering cost calculators while outlining architecture EC2 Cost Calculation: Determine the cost per hour for the selected EC2 instance type. Let's assume it is $0.05 per hour. Multiply the cost per hour by the number of hours in a month (720 hours): $0.05/hour * 720 hours = $36 App Runner Cost Calculation: Determine the App Runner cost per hour. Let's assume it is $0.07 per hour. Multiply the cost per hour by the number of hours in a month (720 hours): $0.07/hour * 720 hours = $50.40

[@nonya3259]

Agreed

[@NathanHedglin]

Exactly! Drives me nuts. The Prime Video blog shows even Prime Video devs don't know what they're talking about

[@yessopie]

He also seems to think that serverless is the only way to have managed hosting. As in, if you have dedicated servers, you have to install/manage them yourself. But there are hosts that do managed dedicated servers with 24/7 support. They can even use puppet, or similar software, so that you can have phoenix-servers without dealing with docker.

[@MosiurRahman-dl5ts]

Tom is a genius, he could've scaled it much better.

[@zeuglcockatrice4633]

he could have scaled it diagonally

[@shadamethyst1258]

He would use svn to share the pictures across the workers

[@zweitekonto9654]

who tf is tom?

[@kartik4792]

@zweite Konto Tom is a genius

[@joshjackson5274]

@@zweitekonto9654 a guy from an article on stream - I can’t recall it all but something like he made a tool called JDSL and a guy added a comment to it and it broke the whole thing

[@dadbod591]

"that doesn't look like a high school project" 💀 I'm a frontend engineer at Amazon and this is so true. There is no culture of frontend development at Amazon

[@nieczerwony]

What's the point? If something is working and brings profit (a lot), why change it?

[@dadbod591]

blockbuster circa 2002: "we're bringing in profit (a lot), why change it?"

[@sarethdarva]

Hey fellow FEE-I don’t think Amazon has ever had much of a front end culture though I see efforts to change it. (Like hiring people with our job title…) and if the amount of time and discussion it takes to make UI design changes I’ve seen is typical it’s going to take a while… Then again I’m not enamored of the UI for Google either. I think you need a design-first ethos like Apple really to have the exceptional UI/UX-and even that has its pitfalls, like when Jony Ive ran unchecked for years.

[@liquidsnake6879]

That tells me that there's no FE leads at Amazon, you need someone to take the lead, to assume responsability, to go up to face people, make themselves public figures within the company to challenge standing paradigms, and it's invariably a problem if you're all just the kind of people who are content in your home office coding whatever they tell you to code. Nothing changes this way, to change things you need people making themselves known and pushing for changes. Admitedly in any given company, very few people (i'd even say 1 in a million) do this because it's an enormous pain in the rear in a big corporation, you'll get a lot of pushback, you'll need to join a lot of extra meetings, you'll need to take on a lot of extra accountability, all while researching and studying to make sure the next architect or designer you get in an argument with doesn't humiliate you publicly on the company chat app lol (and they will try because they will resent you if you're pushing back against their status quo) "But why don't the architects do that?" From my experience in a medium-large (300+ devs) corporation it's because they were all drafted from the Backend development circles and don't care about FE their idea of architecture is much more focused on the links between BE microservices (which they adore and create thousands of) and various hosting and caching services, FE is always an afterthought to them Most FE architecture guidance i ever got was that they wanted microfrontends we needed to integrate with s then they wanted webcomponents instead, then they wanted us to bundle every micro frontend in a monorepo, now they want us to switch to using Github's CI instead of Jenkins etc it's a confusing mess. But there's always a group or taskforce that's seen as the "core", that might be the people in charge of your css framework libraries, might be people in charge of some core packages that everyone uses, a good idea if you want to change things is to try and get into those taskforces or teams, it's easier to change things from inside those teams as you'll have a bigger voice and more people will listen. But overall it's awfully political to change things in big companies and very few people are cut out to do that or even care enough to go through the hassle

[@jel1951]

I imagine because the market Amazon appeals to through AWS isn't concerned with the frontend, rather just the functionality.

[@onee1594]

Problem was not serverless and not even microservices. Problem was bad design

[@Ivan26t]

Amazon needs to hire UX designers and front-end developers. Amazon's website was built by a backend developer 😅

[@mennovanlavieren3885]

They did, they are working on QoS monitoring.

[@BusinessWolf1]

@@mennovanlavieren3885 lol nice

[@funmbiie]

Word!

[@cristi724]

Then it would just look like youtube but also run like shit.

[@kuhluhOG]

not to dump on UX devs, but most of them made UX worse of basically every website I know of which had a decently big-ish rework in the last 5 years

[@TechBuddy_]

Tom is a genius

[@TheThunder005]

"You got to go Because TOM IS A GENIUS!" Prime "How do you spell gene-i-o-s" Tom "you're fired!"

[@TechBuddy_]

@@TheThunder005 Akshually you are wrong TOM IS A GENIUS!

[@fallenIights]

Who is tom? Serious question

[@TechBuddy_]

@@fallenIights Tom is a genius!

[@fallenIights]

@@TechBuddy_ tom is a gay

[@TECHN01200]

It kinda makes sense, you don't pay the internet bill if you don't transfer anything over the internet.

[@OBGynKenobi]

Pretty soon we'll all be programming in assembly again.

[@poopoo-dk4hu]

Already do 🤣

[@zweitekonto9654]

@@poopoo-dk4hu name checks out

[@NachitenRemix]

We go full circle every couple of years lmao

[@IvanRandomDude]

And we will claim that it is incredible innovation and game changer

[@BusinessWolf1]

and we'll be better off for it

[@DeusExAstra]

I think if you just run your micro services on Subversion, that would solve all your problems. Also, Tom is a genius.

[@ThePrimeTimeagen]

Hey, I know you know, but not everybody knows how much of a genius Tom is

[@pinklemonade8444]

​@@ThePrimeTimeagen watching you without facecam would be funny

[@bluecement]

Lmao at DHH bit

[@chrisE815]

"Someone call a data scientist in here immediately." Said no one trying to get something important done...

[@bernardogalvao4448]

Im one and youre right

[@dandogamer]

Data engineer would be the correct role here

[@IvanRandomDude]

Tom can fix serverless

[@pietersmit621]

Tom will JDSL the shit out of it.

[@yuriib5483]

our devs recently accrued 200k spend on cloudwatch, over weekend, after Friday dev branch change because they looped lambda calling itself and their code had verbose logging. with 10mil monthly spend it is drop in a bucket but for small org that's be end of company. most dinosaurs of development don't understand cloud infra and cost pillar well enough to be doing serverless

[@adambickford8720]

In practice AWS will likely forgive that

[@yuriib5483]

@@adambickford8720 just like with cryptominers it is on account owners to pay attention to cost. it is a legit pay as you go storage usage so there is not wiggle room like with let's say ddos refund via shield advanced.

[@BusinessWolf1]

@@yuriib5483 You're vastly underestimating how willing amazon is to wiggle, but that's because you're stuck thinking officiality is real. I'm 100% certain that they would refund the company if they asked. But nobody asks, because they think like you.

[@yuriib5483]

@@BusinessWolf1 15 mil a month bill, shaved off another 400k by pointing out bug in RDS pricing model, we did ask and it is leging cloudwatch storage usage. THey might be lenient at legit hyper charge mistake by some tiny startup so that they don't go under. with big orgs? not so fast

[@dmitriylevy7865]

3:35 that laughter is legendary

[@martg0]

Serverless is ok depending on the use case, not for everything. I’ve done migrations and switched from a 3500 USD/mo bill using ec2, and RDS to less than 100USD/mo using nextjs running on lambdas + DynamoDB + Cloudfront

[@mattmurphy7030]

I’m using lambda and dynamo to run the backend of my game, for player profiles and stats and stuff, and I couldn’t be luckier that this stuff is available. So far I haven’t paid a dollar or spent more than an hour setting anything up. Huge improvement over 15 years ago.

[@yuriib5483]

so you went from relational to nosql? I would not call this a migration. this is a full replatforming. lambda + dynamodb + cloudfrom +route53 stack also assumes you have a modest usage. I is very easy to run away from your cost projections if you have bad actors hammering your app. hope you got that shield advanced

[@dandogamer]

@@mattmurphy7030 lol the latency is going to be insane, lamba for a game?

[@mattmurphy7030]

@@dandogamer read it again

[@martg0]

you are right, we rewrote all the app, and avoided a DevOps team too. You are right regarding bad actors too, we implemented WAF and api rate limits. And now identified silent errors on the client side using Sentry.

[@rubyh4184]

The key of designing system that could scale horizontally is Stateless. Monolith could scale horizontally as long as the instances are stateless. The answer was never always microservice to scale horizontally.

[@user-jj2tr4pi3z]

The world is not ready for serverless JDSL.

[@wlockuz4467]

The main point is that Amazon Prime was doing something thats inherently impossible with Serverless and thats how they save 90% by moving away from it.

[@Zzznmop]

I thought the main point was that they coupled two microservices to share a data store

[@bravethomasyt]

They could have saved the 90% without even moving away from serverless.

[@gliderspace]

I used to write algorithms to do something kind of similar. We would turn videos into frames and check them for big sections of similar colors and turn them into matrixes to check how close they were to solid color squares (common video encoding error artifacts) in order to validate if the movie could be sent do Netflix, Google Play, Apple TV, etc. It ATE our resources like HELL! We had dedicated HP workstations with two quadro boards each just to do that work (and a few other validations). It was HEAVY!

[@jefffriedman6942]

"You'll compute nothing and you'll be happy" - LOL

[@Hyrum.]

JDSL definitely could’ve handled it. Tom is a genius! Could’ve deployed the whole thing in comments.

[@ninjaasmoke]

ugh aws lambdas are just ugh. i recently joined a company that has a micro service’s architecture built with around 1000+ lambdas. each doing different things. for each, we have asked aws to allow upto 20k concurrent requests. now we are supposed to handle over 40k as our company’s growing. and the time AWS takes to scale the lambda is over 30minutes!?? we are switching to kubernetes jobs as an alternative. and now i’ve to rewrite code for 1k lambdas 🙂

[@HobokerDev]

Job security 👍?

[@yuriib5483]

not much to rewrite even if you're running something exotic like .net3.1, or nodejs12. most likely will just run out of the box. if lambda runs it then there is most likely container image for that too. we got 140k lambdas and some developers have died during 'Rona and no one got time to read the code, and some were edited 5 years ago, now that's some danger zone

[@Nathan00at78Uuiu]

How dare you question the genius of Tom. He literally built jay diesel by himself with executable comments at runtime.

[@jogurtnaturalny]

Tom is a Genius

[@thegrumpydeveloper]

They spent so much time asking if they could do it instead of if they should do it.

[@dipereira0123]

11:59 Goddamn years following Primeagem and never ever imagined that he could be a man of culture

[@sunilmadan2006]

Prime issue was S3 bucket read / write cost and not serverless cost.

[@thebodachshow]

My work uses one single dedicated server (well 2, we have a mirror fall back) costing £120pm each, and it's running a collection of dockerised microservices. it's cheap, cheerful, low maintenance and scalable vertically and horizontally. Currently handling around 350k requests per hour, with file uploads, file processing including video etc. With Cloudflare caching for static files, which they do for free, it's so easy and cheap and "future proof" to go with an SOA approach. Our system on AWS or edge would cost £thousands per month... embrace the dedicated server

[@stevenhe3462]

This stream duplication business is no surprise when you see "ML" in their workflow. Probably, some data scientists planed out this mess.

[@Nick_fb]

Serverless is a gamble, but Anakin memes are forever

[@dmitryplatonov]

I think you are wrong about what they are doing at amazon: they re-stream streams from _clients_ to qa service (or do snapshots at clients and upload, not sure). And they do it not for every client, just for some small subset. So no additional load on streaming servers.

[@riptorforever2]

04:02 mr vim, when you hood drop off, was like a person loosing the ears, hair, humanity... this hood is just part of you. now I understand why superman could disguise to clark kent by putting a glass

[@FreddyDaleginds]

it's a far above my hat show, but I made notes and am going to think about it in my head. I send you a message a while back about how you inspired me to change careers but something is holding me back it feels, so I resuscribed to your channel :)

[@leetaeryeo5269]

“It went from crazy-ass to… less crazy-ass” It went from crazy-ass to crazy-elbow

[@filiformis]

"for some reason it's like it's 10x cheaper to not have 10 streams." lol lmao, even

[@cookiesInChocolate]

I think Cloudflare Workers is a great example of lightweight and cheap “functions”.

[@jordixboy]

Hey prime! im curios, you use vim as editor, do you ever use a debugger? what debugger?

[@clarkm]

I don’t get why everyone is up in arms about this. The promise was never that you could fragment your application into infinitely small pieces and it would magically be better. There are certain parts of an application that cannot efficiently be divided up into micro services, and there are others that are perfect for it. I don’t understand what the revelation is here.

[@PetterAmland]

Oh, so that's what serverless means. I thought it meant there were no server, like a normal client side only program

[@BusinessWolf1]

uh... WHAT.

[@RaevGames]

Tom is a genius, he would've listened to Moonlight Shadow

[@MrTiteuf12]

Wasn't expecting that Thomas Sowell quote, pure gold

[@SteveBriley]

That "Ruby on Rails" programming without types is vastly superior bit. ROFL. I'd offer to give them my footgun, but it's clear they have more than I do.

[@TizzyD]

NEVER SAY NEVER. Serverless CAN BE good. It depends on the data interdependence and flows, storage, etc. Additionally, there are many business inputs to consider: resource availability, on-premise resource utilization, etc. Yes, you can save money by self-hosting...but...the money is paid elsewhere. Reminds me of a search challenge I had 20 years ago. The query had 7 functions to transform the returned data. How to speed it up? Pre-compute the transformations, e.g. denormalize and prep asynchronously. The result: 2 min queries became 4 sec. (But hey, that was 20 years ago!)

[@Waitwhat469]

At a certain point in scale self-hosting cloud serverices, like serverless hosting in order to reduce cloud cost, but scale out to the cloud as needed.

[@willhleucka821]

The runtime executes comments like normal code!

[@ChrisWrightGuitar]

It's like the clean-codification of deployment. They wrote a load of over-encapsulated functions and tied them all together.... over a network 🙃

[@nnnik3595]

It wasn't even a network. It was a network drive.

[@LegitBanananas]

i love how tom keeps coming back into these videos. I bet its the same Tom from myspace :P

[@jandorniak6473]

Wait, wasn't edge computing the new on-prem? In a hybrid model?

[@Tomplaysguitar182]

Unless you're deploying your jdsl on the edge, you're an imposter.

[@Guergeiro]

Title should've been: "How I solved Amazon Prime Video problems in 45s".

[@osarogie]

6:05 The Lord's answer to all of our problems 😂

[@Yupppi]

"This is an unironic real picture of Netflix" made me feel so sorry for him.

[@Griffith-Did-Nothing-Wrong]

amazing that AWS didn't even follow their own recommendations for serverless applications. A 1) long-running process with 2) Consistently high workloads? Nothing can go wrong here!

[@disguysn]

AWS doesn't coordinate efforts across the company incredibly well from what I can tell. They do it much better than virtually every other company out there, but it's still not that great.

[@diegolikescode]

Broh, they didn't use AWS Calculator

[@boomerau]

Most people in tech just do it and don't worry about the cost. The business also know better and tell you to do it if you do ask. It is always interesting to watch the business buck pass their solution around.

[@nnnik3595]

One of my senior dev colleagues kept telling me that "costs are unpredictable" and "decisions shouldn't be made with costs in mind" The next project we build now cost the company 17 cents per month in serverless stuff.

[@JacobHo161]

3:02 "And still be unable to make a wesite that doesn't look like a high school project" I also wonder if they even have a UI designer.

[@iulianbojinca7309]

"Beyond stupid" - Linus Torvalds about AWS engineers...

[@hotrodhunk7389]

3:30 Amazon prime is what makes them successful. Next day shipping for no added costs. Billions of products at good prices.

[@yuriib5483]

not really, cheap chinese garbage and being one of the very first on market. now practically every store has some options of free shipping

[@jonforhan9196]

I’m going to re-write all of libavcodec in typescript, Tom’s a genius

[@Alex-hr2df]

The best solution is to adopt a hybrid approach that combines serverless and monolithic architectures. Critical components benefit from serverless scalability, while less dynamic parts are consolidated into a monolith. This maximizes resource utilization, cost efficiency, and flexibility in scaling. Continuous monitoring and optimization ensure a cost-effective and scalable solution.

[@Jabberwockybird]

At the end there. That sounds like a national anthem from The Critc. Eee pii tii pii pee pee pou pii

[@leakyabstraction]

The funnies thing about Amazon is that it feels like a clueless booksalesman is trying to sell me a bunch of stuff when I use their site. "Oh, you bought a bed last week, would you fancy buying this another bed as well?" "I can see that you're searching for RX 7900 XT... How about these RTX 3050's instead?" It's a mystery to me how did they get so big.

[@badpussycat]

LOL Never understood why people are so excited about those recommendation engine. It really sucks!

[@ryanquinn1257]

I diiiiiiiiiiiiied at that Amazon rant about being like oh cool I want to watch Conan. Wait I have to buy it? Screams externally and internally*

[@ivanmaglica264]

Stream and S3 - how does that go together? Using S3 for realtime applications seams to me like taking aeroplane to go to shopping mall.

[@adambickford8720]

Serverless is a gamble: will you make for up the rip-off $/GFLOP of compute with your actual duty cycle? 10x the cost but 1/20th the usage is a win. Of course aws knows that by the time you hit that breakpoint you're probably locked-in :evil-pinky: Remember, aws is renting you the same hardware in Virginia either way.

[@kdaniels98]

I love the callback :)

[@SurfsUpSeth]

I’m convinced Amazon makes their AWS website and documentation look terrible so that people have to pay for their support service

[@azotic1]

Is it possible that AWS had a mole in Prime Video who was attempting to run up the AWS bill?

[@dingo35]

This guy just “does not know”

[@doctorstal]

Amazon builds sites like that so customers think they are buying cheaper stuff from those sites. It IS their success strategy.

[@morgengabe1]

Netflix might have a prettier website than Amazon but come on man, Netflix's ui is so minimally useful you should know better than to throw them kinds of stones

[@Chris-jm2gx]

Hearing J.Diesel gives me Vietnam War flashbacks, and I'm not even involved.

[@user-yo2wt7xw8e]

3:41 i need a time selection for this!

[@beto5720]

TomGPT would have suggested a monolith right away

[@8koi245]

Chat sharing a single braincell AYAYAYA

[@pauldraper1736]

2024 is the year of serverlessless

[@John-3692]

I'm blown away by this content. I had the privilege of reading something similar, and it was truly incredible. "Mastering AWS: A Software Engineers Guide" by Nathan Vale

[@reveil85]

Serverless only makes sense in case when your usage pattern is extremely uneven. One example is when you have a computation heavy report to do once per month. If just you rent servers to do it they sit unused for 29/30 days. Unless you are using them for something else in the meantime (development/CI etc) then serverless is a valid solution for this. On the other hand If you just have web service in 90% of cases avoiding serverless would actually save you money.

[@MacMiggity]

Who is Tom? We talking about Tom from MySpace?

[@DigitalNomadOnFIRE]

Google is also incapable of making a nice looking website.

[@TheDorac1]

I come back here to watch 3:40 when i'm having a bad day

[@officialabdulrehman]

11:58 loooool

[@whig01]

I feel like microservices could just be served from Git with TOML configurations to execute on each request.

[@BusinessWolf1]

internet traffic costs

[@chrisE815]

You're the master of edging...

[@KadenCartwright]

Tom would’ve piped all the video streams into SVN instead of using S3

[@asdqwe4427]

Amazon’s websites look like shit for the same reason that the Dollar store looks like shit. I Tricks people to think that everything is cheap

[@To1ne]

Pretty sure @Fireship planted that DHH just for The Prime to respond.

[@andy_ppp]

The AWS Console often doesn’t even update properly let alone look decent.

[@ersetzbar.]

what are you doing step-function?

[@user-qr4jf4tv2x]

edge based computing is only good when you don't hit the database or was tailored to that edge otherwise you are making it worse because you might have moved your servers closer to the user but now you have traffic issue to the database.

[@badpussycat]

in that case, you have a distributed database with a cluster close to the edge

[@airman122469]

I almost always prefer self-hosted monoliths. They’re almost always simpler to maintain and given enough traffic are cheaper to run. Though, it’s possible I’m biased because the micro service’s architectures I’ve seen have always been inefficient piles of garbage.

[@thatmg]

Tom, The Savior.

[@micycle8778]

12:00 reject nightcore-chan and embrace drum and bass waifu

[@ingloriouspancake7529]

Someone redesigned the Amazon app and they offered him a job

[@n0madtv]

It only costs less at the beginning, there are diminishing returns to any cloud service. It's just the nature of 'middle man' economics. Cloud services (all of them) are great if you're a startup and can't afford 100k worth of servers just to put in 2 locations that will cost another $1k a month to operate. But once you have scaled cloud services up to $1k a month, or beyond, it's time to start putting that money towards financing the ownership of those servers. Or, better yet, by the time you've scaled to $1k a month cloud services, maybe you've already made 100k??? I know of a single game server that hosts around 1k users at any given time being hosted on AWS (bare metal), and it's something like $10-20k per month. And they don't see a problem with that because they're all devs and don't want to deal with servers. Hire me for $4k a month and finance some servers! We'll both pay for ourselves! Also, anyone familiar with the game Star Citizen will know about all the problems they've had the last few months. They are basically refusing to scale their services and just deal with poor performance. But, here's the kicker, they are 100% locked into AWS because of early sponsorships and committed discounts. They are probably spending over $100k a month (my personal estimate). They could be putting redundant servers in a new region every month for that price and then pay the $10k a month to keep 10+ locations running (plus pay 2 people $100k a year to manage them). And this is a fully funded operation with 10's of millions in cash...

[@fuzzy-02]

Bam!

[@its_finn96]

True that on Amazon’s websites being total ass

[@sabrinapyles563]

since so many other companies pay to throw a front end on aws, and then even pay aws while offering a free tier to users of the service, amazon is almost...incentivied to have dogwater front ends?

[@stealtime]

curious to know what a "v math" score is, or whatever it was that @theprimetimeagen was referring to at the end there

[@haloboy777]

I agree Tom. Is. A. Genius. 👏👏👏👏

[@SpiraSpiraSpira]

Our company saved something like $10,000 a year just by buying an old 2U Supermicro Xeon server and spending a couple hundred bucks a month to co-locate at an ISP. We still use some AWS stuff, but only as overflow scaling.

[@KrisMeister]

It depends on if you get large spikes in traffic. You'd need to buy, load balance and maintain more servers even though they're mostly not in use. That is the use case for cloud and lambda. - like you said.

[@jasonwhisnant5457]

I’ve seen MiMo a lot on his KZbin screen after playing vids. Is he a MiMo fan?

[@avi12]

3:52 Live footage of Primeagen losing his mind