Two Weeks Of DDOS Attacks - Did We Survive?

Thank you Vercel for helping us survive this one 😅
ALL MY VIDEOS ARE POSTED EARLY ON PATREON / t3dotgg
Everything else (Twitch, Twitter, Discord & my blog): t3.gg/links

[@t3dotgg]

CHECK OUT VERCEL AND UPSTASH BECAUSE THEY KEPT US ALIVE THROUGHOUT THIS vercel.com/?ref=theo upstash.com/?

[@dasrite]

Am i correct in presuming the only reason you're not gonna be billed for this is because you're sponsored? So if it was me i'd be getting billed for a DDOS attack? that's enough of a yikes for me tbh lol i'd rather it crash than it staying up and them charging me tenfold lol

[@t3dotgg]

@@dasrite no, they would help any customers going through this

[@Hexalyse]

Either they bought botnet time and it costed them money directly; or they used their own botnet (instead of renting it to other people during this time), and that's money they could have earned but didn't. So the result is the same : it cost them money. Probably a decent amount. Which I find hilarious considering the ridiculous impact they had.

[@sardines7436]

not to mention theyre giving him monetizable content lol. from his pockets to theo’s

[@Hexalyse]

@@sardines7436 and good publicity for Vervel too, seeing how easy it was to handle the problem with them. Conspiracy theory plot twist : it's actually Vercel themselves who conducted the attack so that Theo make this video to further PR their services to his followers, for free.

[@Manas-co8wl]

@@Hexalyse i'm glad i'm not the only paranoid one who thought of this. i even went a step further..

[@jordixboy]

he gain internet fame thats better than money

[@sajanah1253]

Just curious, how much it could have cost for this attack?

[@somedick7337]

We had a DDoS attack about a year ago where it was about ~10TB/minute and we are hosted behind cloudflare, so just couple clicks inside cloudflare panel (there is a button "we are under attack") and this attack is gone, next minute I checked where is it comes from and every single IP of attack came from outside of my country (No one wants to ddos from same country or your business since police could investigate it and attacker could end up in jail), so I did just enabled captcha for any request form outside of my country (since our business doesnt have international customers) and disabled "we are under attack" and never had an issues since then while they still trying (one year later). So may be something like cloudflare could help you.

[@T42nk]

You should make a dedicated video about DDoS-protection on the T3 stack, as clearly there is a possibility of creating unreasonable cost for the service provider and not everyone will have their bill refunded. You mention as a side note that you could put rate limiting on each route for a personal CloudFlare, maybe you expand on that and/or provide a package as framework for that.

[@emanuelfarauanu1760]

Who has that much time to waste on a DDOS attack that gains them absolutely no benefits? It's clear that your tech stack handled the attack quite well, it didn't even cost you much at all. If you get enough views you'll probably even get money out of this video that they helped create. Someone really hated you, lol.

[@NihongoWakannai]

For some reason there are random tech nerds who REALLY love to die over tiny molehills for no reason

[@augustoeduardo209]

cant understand why someone waste his time to do that....

[@josemfcheo]

Maybe written in Rust...

[@brahimbenfares1464]

Apparently aws didn't like that video lol

[@milanpatel3159]

@@josemfcheo beware of usage of that word bro 😂 (the R word)

[@EddyVinck]

This video got hundreds of hits in the first few minutes. Maybe your KZbin is getting DDOS’d 😮

[@emanuelfarauanu1760]

KZbin alg really loved this video, it showed it to me in the recommendations the minute it was published. That was with me not watching many of the previous videos.

[@sebaarnio]

This happened to me too! It might not have been targeted towards you because it happened to a test deployment of mine which didn't even have any real traffic. It was also on Vercel, and I get blocked pretty quickly (although support helped me get unblocked) Edit: My attack caused 462 GB-Hrs within like 20 minutes

[@williamx0]

The key is that you need to rate limit these attacks whether you use server or serverless. So this demonstrates serverless has tools to handle it. Ofc if you had a server and rate limited it could handle it too. Re one of your last statements, it doesn’t mean servers would’ve been worse. It means not rate limiting would’ve been bad, server or serverless

[@hhhhhhhhhhhhhhhhhhhhhh]

This was not even that big of an attack. The traffic is literally less than 1 GB per second. If anything this was a skid attack which is further supported by them literally just loading one JS file over and over. This wasn't a DDOS attack, this was some kid trying out their $5 booter.

[@OfoeNelson]

So this is not on the same level as the ddos attack that took down google

[@lev2590]

Considering that one 10gbit server can (in theory) handle 1.5tb raw traffic (in 20min) I don't believe that this was a big DDOS attack. Also I believe that having multiple cheap vps with high bandwith automatically deployed when needed, would be probably way cheaper than vercel. Obv. the developer experience will be worse, especially when setting all the servers up or other cluster related issues occure.

[@perc-ai]

Yes this was just 15 yr old some kid with a very small botnet. A pro would have used 10k IPs and Vercel would have to shutdown their dns for a period of time

[@dandogamer]

If you weren't on the pro license and sponsored by vercel this might have been a different story. I can imagine a normal person would have to suck up the big fees or take down their application

[@mormantu8561]

It depends. Cloud providers more commonly than you might think cover ridiculous fees in case of an error or attack like this. Because they don't want to lose your business.

[@Knightfall23]

They’ll most likely cover it the first time but if it happens again your on the hook

[@mormantu8561]

@@Knightfall23 Agreed.

[@dasrite]

@@Knightfall23 I'd rather get a straight response from Vercel directly than try to imagine what might happen

[@samdcbu]

If you are a normal user on the free tier of vercel you would just put your application behind Cloudflare for free and let them handle the DDoS traffic.

[@Shri]

It is actually really cheap to buy residential proxies (pools with millions of IP addresses) and then use them to bombard requests to services. These residential proxies exist to enable scraping of SERP content as well as regular sites with hardened DDOS protections. Some residential proxy services also bypass recaptchas (using AI and sometimes even humans) for premium. Residential proxies have legitimate use cases but can be misused to create botnets too. That is what I am suspecting is happening here. They haven't actually paid for those 600 IPs. Rather, they are tapping into a pool of million IPs provided by residential proxy services.

[@NiSiRewinD]

It's free, even on that scale. There are a ton of private projects for geting around state cencorship, x100.000 of IPs, since the Ukraine war shined a light on how those countries block access to Tor itself. No idea how big those requests were, but generating a couple Gbps traffic is still negible, as far as paid botnets would go. Those networks typically limit access to specific websites, but I guess it wouldn't be crazy to get around that, as user, depending on how they filter. I suspect this didn't run via Tor, but I guess we would with a list of the IPs

[@ludawig_]

I am glad that there is an official report, from the DDOS Foundation, on this incident and that Theo is now a part of it! 👍😉

[@TheLKStar]

I mean, having a punchable face and arrogant personality is bound to provoke someone when exposing yourself to thousands of strangers. Even so, it takes some extra thick emotional issues to waste any amount of time and resources to get revenge on a parasocial relationship.

[@unnaturalatrophy5377]

Damn dude 😂

[@TheJulsMan]

You got lucky because you are publicly sponsored ... but this brings up a BIGGER point. When you pay for metered services, the providers NEED to indemnify you against DDOS attacks or other potentially ruinous events. This could be a huge selling point, because not all services have built in caps. As a matter of experience, I witnessed a peer who used AWS, his application while still in beta had a memory leak and AWS sent him a $13k bill. Insane! Risk avoidance is important to any business especially if there is no ceiling or price cap. As someone who uses these services this keeps me up at night.

[@hanes2]

I think both CloudFlare and Linode (Akamai CDN) has ddos protection included.

[@canht95]

It was probably @theprimeagen...

[@jocdiazm]

Love that the stack you are recommending is the one that you use for your stuff. I can imagine that the people behind this were just absurdly annoyed that you are recommending tools that don't fit their certificates or what they consider is the "right move".

[@AJ-wf1vh]

huh, so vercel has no rate limitting by default? I would have expected a managed service to handle this, not have me set up my own edge middle ware (upstash?) It doesn't look good on their part

[@callowaysutton]

Did Vercel give more details? Such as if the IP addresses were all from the same IP block or dispersed across many, whether or not they were residential IPs, their own IPs, IPs from other cloud providers, etc, geolocation lookups of the IPs? All of this seems like it would be super useful to know about to prevent future attacks both for you and them

[@bluesteelbass]

Willing to bet those IP addresses got put on one of the many naughty lists that are distributed to rulesets for firewalls.

[@alitonoliveira1700]

It would be great to know how to prevent a DDOS attack against AWS and GCP (Cloud Run and Cloud Functions).

[@QckSGaming]

GCP: Toggle the DDOS shield on. Cloud Armor it was called I think.

[@benheidemann3836]

Love it, keep them coming

[@TheGetawayMan]

Side note - The vid I'm most looking forward to is the one you mentioned about syncing clerk with your own db 🙂

[@ilijanl]

Is it me, but why are they are targeting static assets? I mean if you want to increase Theo's bill, DDoS the api route which has the upstash rate limiter as well? It will cost him 0.20 cents - 0.40 cent per 100k request for upstash and probably far more for serverless/edge functions on vercel. Thus sending 100 milion request will at least cost 1000 * 0.20 + 500 GB hours ( 5*40) = 400 dollar + rest of vercel

[@perc-ai]

Tbh this attack probably came from a 15 yr old… this is not a serious attack tbh it’s very easy to rotate 10k residential proxies and force vercel to temporarily shutdown all ping services I could probably do it

[@Fervore_]

at what point does vercel consider the requests as a ddos attack do they use any tools? what happens if a tiny dev's app gets ddossed, would vercel refund 100% of the money by all requests that day? how long do they take to answer from the point where you're under attack to when the situation gets resolved?

[@nickwoodward819]

after 1 minute according to their webpage. not great really given the number of requests you could be on the hook for at that point. and no, if they don't say they'll refund your money, assume they won't.

[@_va3y]

>>> I think it’s quite impossible to take us down with this stack Bro, you’ve just broke the main rule of opsec 😅😅

[@Mitsunee_]

if you're putting the upstash ratelimiter infront of everything, how are you not hitting ratelimits on upstash?

[@lightninginmyhands4878]

Where is redis in your data fetching flow?

[@stevenismart]

Can you do a more in depth video on how to stop DDOS and other random attacks?

[@Rensoku611]

6:09 I dont understand this part. Why would you fare worse if you had actual servers? They have rate limiting and IP blacklisting as well

[@jacoblockwood4034]

I think he’s saying that if they were hitting the actual servers, all the endpoints would be destroyed pretty quickly

[@Daddyjs]

That would be an awesome tutorial setting up ddos protection using upstash

[@eleldevelop-bu2md]

Can someone describe the AWS scenario? What would have happened and how to do rate limiting?

[@lapulapucityrider3227]

Use ec2 not serverless much better

[@masseeerra]

Would be good to get an in depth video on the specifics of how you (or the tech) delt with it 🙏

[@yunyang6267]

I have a question. Are those people generating these attacks going to be held accountable or there are ways to generate botnets attack and get away with it

[@dhkatz_]

Can you not put a Vercel site behind a Cloudflare proxy?

[@CoderandFilmmaker]

Honestly Theo, now I am very relaxed about the decision of using T3 Stack and the services you recommend us. If even Chirp handled this insanity! Then we’re in safe hands as Solo-preneurs 😊

[@lescobrandon2202]

Good stuff.

[@ZukoThePrince]

your hair looks majestic dude !!!

[@user-tz6nn8iw9m]

ddos a static file :genius:

[@jmarbutt23]

Where is your video on the rate limiting with UpStash?

[@udaym4204]

upstash rate limit is good for backend but what about fronted to save from ddos attack

[@maddsua]

Yep, serverless rocks. Won't ever go back to dedicated hardware

[@codebelb4767]

Jeff Bezos hired DDOS assassins to protect his stack

[@socialkruption]

Wouldn't surprise me if some from /g/ were part of this, your videos have started making thier rounds over on the board.

[@t3dotgg]

Oh really? Fuck yeah finally they're gonna HATE my ass

[@hyper_channel]

/g/ cares about nothing but LLMs right now, highly unlikely

[@varma8669]

What is /g/?

[@hyper_channel]

@@varma8669 4chan's technology board

[@g-rexsaurus794]

LLM?

[@mike110111]

How much would it have cost? Isn’t the point they were making is that on demand computing can make your costs sky rocket?

[@thejonte]

Wow! Holy crap!

[@pedrofelipefonsecaenunes2435]

This won me over!

[@brod515]

I can't imagine that someone decided to waste any significant ammount of money doing this. I'm wondering how they had access to 600 static ip addresses.

[@gemmaatroxxibox5322]

DDOS is my main concern with Serverless. With an nginx proxy you can get sub 1ms 503 responses in a DDOS and cap the number of requests per IP so that it doesn't touch your actual app code when it happens. So for a free or cheap server vs a Serverless platform, at least getting started out, I know someone can't rack up costs for me.

[@rifwann]

dude, im sold on this platform you are using.. what a way to advertise..

[@markclynch]

It's laughable how much the attacker likely spent vs what you incurred. Perfect example of mitigation, you can't stop it from happening, you have to make it too expensive for bad actors to continue.

[@name_less227]

Don’t these type of attacks usually use hacked computers to help them attack?

[@Hexalyse]

@@name_less227 They do. It might not "cost" them anything in the literal sense. They didn't spend money most probably, if they own the botnet. BUT... usually when you own such a botnet, you can sell it, or rather rent it to people who want to conduct such attacks. So all in all, either their bought botnet time and is cost them money directly; or they used their own botnet instead of renting it, and that's money they could have earned but didn't. So the result is the same : attacking cost them money. Which I find hilarious considering the ridiculous impact they had.

[@jason_v12345]

But you can stop it from happening. Rate limiting middleware can be written in a just a few lines of code.

[@perc-ai]

@@jason_v12345 Theo doesn’t even know what he is doing lol

[@dueft4479]

How much did it actually cost you?

[@hseinb]

How do you know the attackers have used static IP addresses?

[@filda2005]

if it did not chang within 2weeks, it is not so much dynamic IP

[@andriiantoniuk8419]

But, what exact cost for you of this DDOS attack?

[@sortof3337]

What do you think about IP based rate limiters? Would they prevent such attacks? That's the only protection I have against DDOs.

[@ttrss]

ddos atks are (Distributed)DOS

[@sortof3337]

@@ttrss so its like wearing a hat to be bulletproof. Nice.

[@ttrss]

@Sort of 😄i guess. And then cloudflare protection is like letting a government protect you, but they're like super authoritarian.

[@sandrinjoy]

we got ddosed, and vercel did nothing. 6.8TB

[@xavhow]

Just wondering why it’s costing the attacker more? The hundreds of IPs used by the botnet are probably someone else’s.

[@SeanCassiere]

Does anyone have any examples of using upstash's rate-limiter with tRPC? Been using it more, and I haven't really gotten around the concept of how rate-limiting could be added to it. It most likely would be done via a middleware, but just putting up the flag for any existing repos that have it.

[@liam.brewer]

he covers it in his newest t3 stack course where he builds a twitter clone called chirp

[@SeanCassiere]

@@liam.brewer thanks! My bad for not fully checking the repo.

[@Joseleon-ct8xz]

it migth be done from the rust foundation

[@jventura1738]

Theo makes a video about Twitch dying then gets DDOS’d hmmm… 😂 all jokes aside your services handled well!

[@just_ppe]

Nice promotional video :-)

[@mike110111]

Would a simple solution be - count how many requests for each IP address, and if it goes above 100 per second you block them?

[@shahkaleem6601]

Because you were Ddosed you have got a subscriber.

[@JuanRodriguez-rh4kp]

It was probably the angular team trying to make t3 look bad.

[@MrLT-vf3wr]

Who has the resources to pull this off? Amazon does, that's who...

[@foswa6335]

Theo what are these thumbnails 😂

[@greendsnow]

1400 GB costs USD 180 on Vercel. (First 1000 is USD 20) What're you talking about?!

[@rizkiaprita]

i see what you did there

[@jbbzzi]

Absolutely hilarious that some people will spend thousands out of pure spite for absolutely nothing. Still, I can't help but be skeptical of these new cloud providers you're showcasing. When the 'growth' period ends and the 'taking profits' period begins, is it still going to be more attractive than AWS? We'll see!

[@marcelor1235]

What contents do you have about Vercel? I guess we all have some app to protect.

[@Daddyjs]

How would you be dead if you had actual servers running this? Wouldnt it be better because then you wont get charged a lot of money? Thanks for helping a newb like me understand

[@Knightfall23]

Gotcha

[@GeorgeDicu-hs5yp]

Oh nice!! Long live rate limiting.

[@anasouardini]

Instead of bringing down your services, they just gave you a topic to talk about? hilarious!!

[@spectator5144]

Vercel needs bun

[@RedStone576]

bun on the edge?

[@cowabunga2597]

Pork bun ?

[@nicolasguillenc]

People can be nasty, that’s why it’s better sometimes to build in private. Oh and good for you man the infrastructure really held up

[@venicebeachsurfer]

Did Vercel try to charge you?

[@justmrmendez]

"impossible to take down our services" my guy, you are challenging the wrong community here 😂, and for those that say it cost them money, ego doesn't care about money. We do things some time just to prove that we can, no need to hate you to do something like this (I'm not saying I'm even capable of this), but if i could, i wouldn't do it because hate, but ego or passion, and seems you already have enough of both

[@riolly]

I thought hacker is always smart, they absolutely stupid in this case.

[@FaisalAfroz]

introducing captchas might also help.

[@t3dotgg]

For fetching a JS file?

[@betterinbooks]

this is a pretty cool vercel ad to be honest.

[@JasonJA88]

👍

[@RockRespawn]

oof

[@dog4ik]

Must be rust foundation

[@TomNook.]

Primeagen testing out his Rust pen test code?

[@Khari99]

Well you sold me on Vercel thats for sure.

[@windyace]

lmao they literally gave you content

[@bobanmilisavljevic7857]

Haters are just mad you can beat them in games of skate and they are mongo

[@humansaremortal3803]

If the haters attack again, please have a bowtie for the next vid.

[@headlights-go-up]

Considering how much this cost the attackers and how little it affected you, it had to be someone with disposable income. Seems like Elon wants to get his revenge after you told him how ads work.

[@aryabp]

Seems someone's back-end needs Rust Framework 😊 Edit: idk why my reply multiple times got deleted. so i am sorry, i cant explain due to no freedom of speech

[@IvanRandomDude]

explain

[@NorthernChimp]

@@IvanRandomDude hint: kzbin.info/www/bejne/aKDLaIClett9bpo

[@alessandrosomigli]

This really just shows that TS on the backend can handle this kind of load as well most of the time...

[@t3dotgg]

Comments like this are why it's hard for me to give a crap about Rust tbh - like it's obvious it wouldn't have helped here at all

[@christophersherman8036]

Hope this comment is ironic lol

[@videos40058]

no they didnt burned 500 slots. you burned yourself 500 slots and from all that who knows how many of them was players:))