Thank you Vercel for helping us survive this one 😅 ALL MY VIDEOS ARE POSTED EARLY ON PATREON / t3dotgg Everything else (Twitch, Twitter, Discord & my blog): t3.gg/links
Пікірлер: 157
@t3dotgg Жыл бұрын
CHECK OUT VERCEL AND UPSTASH BECAUSE THEY KEPT US ALIVE THROUGHOUT THIS vercel.com/?ref=theo upstash.com/?
@dasrite Жыл бұрын
Am i correct in presuming the only reason you're not gonna be billed for this is because you're sponsored? So if it was me i'd be getting billed for a DDOS attack? that's enough of a yikes for me tbh lol i'd rather it crash than it staying up and them charging me tenfold lol
@t3dotgg Жыл бұрын
@@dasrite no, they would help any customers going through this
@Hexalyse Жыл бұрын
Either they bought botnet time and it costed them money directly; or they used their own botnet (instead of renting it to other people during this time), and that's money they could have earned but didn't. So the result is the same : it cost them money. Probably a decent amount. Which I find hilarious considering the ridiculous impact they had.
@sardines7436 Жыл бұрын
not to mention theyre giving him monetizable content lol. from his pockets to theo’s
@Hexalyse Жыл бұрын
@@sardines7436 and good publicity for Vervel too, seeing how easy it was to handle the problem with them. Conspiracy theory plot twist : it's actually Vercel themselves who conducted the attack so that Theo make this video to further PR their services to his followers, for free.
@Manas-co8wl Жыл бұрын
@@Hexalyse i'm glad i'm not the only paranoid one who thought of this. i even went a step further..
@jordixboy Жыл бұрын
he gain internet fame thats better than money
@sajanah1253 Жыл бұрын
Just curious, how much it could have cost for this attack?
@somedick7337 Жыл бұрын
We had a DDoS attack about a year ago where it was about ~10TB/minute and we are hosted behind cloudflare, so just couple clicks inside cloudflare panel (there is a button "we are under attack") and this attack is gone, next minute I checked where is it comes from and every single IP of attack came from outside of my country (No one wants to ddos from same country or your business since police could investigate it and attacker could end up in jail), so I did just enabled captcha for any request form outside of my country (since our business doesnt have international customers) and disabled "we are under attack" and never had an issues since then while they still trying (one year later). So may be something like cloudflare could help you.
@SloudPL-oldАй бұрын
It's not a DDoS if there was 1 ip it's a DoS. DDoS you have a lot of ips
@emanuelfarauanu1760 Жыл бұрын
Who has that much time to waste on a DDOS attack that gains them absolutely no benefits? It's clear that your tech stack handled the attack quite well, it didn't even cost you much at all. If you get enough views you'll probably even get money out of this video that they helped create. Someone really hated you, lol.
@NihongoWakannai Жыл бұрын
For some reason there are random tech nerds who REALLY love to die over tiny molehills for no reason
@augustoeduardo209 Жыл бұрын
cant understand why someone waste his time to do that....
@josemfcheo Жыл бұрын
Maybe written in Rust...
@brahimbenfares1464 Жыл бұрын
Apparently aws didn't like that video lol
@milanpatel3159 Жыл бұрын
@@josemfcheo beware of usage of that word bro 😂 (the R word)
@T42nk Жыл бұрын
You should make a dedicated video about DDoS-protection on the T3 stack, as clearly there is a possibility of creating unreasonable cost for the service provider and not everyone will have their bill refunded. You mention as a side note that you could put rate limiting on each route for a personal CloudFlare, maybe you expand on that and/or provide a package as framework for that.
@sebaarnio Жыл бұрын
This happened to me too! It might not have been targeted towards you because it happened to a test deployment of mine which didn't even have any real traffic. It was also on Vercel, and I get blocked pretty quickly (although support helped me get unblocked) Edit: My attack caused 462 GB-Hrs within like 20 minutes
@EddyVinck Жыл бұрын
This video got hundreds of hits in the first few minutes. Maybe your KZbin is getting DDOS’d 😮
@emanuelfarauanu1760 Жыл бұрын
KZbin alg really loved this video, it showed it to me in the recommendations the minute it was published. That was with me not watching many of the previous videos.
@ludawig_ Жыл бұрын
I am glad that there is an official report, from the DDOS Foundation, on this incident and that Theo is now a part of it! 👍😉
@williamx0 Жыл бұрын
The key is that you need to rate limit these attacks whether you use server or serverless. So this demonstrates serverless has tools to handle it. Ofc if you had a server and rate limited it could handle it too. Re one of your last statements, it doesn’t mean servers would’ve been worse. It means not rate limiting would’ve been bad, server or serverless
@hhhhhhhhhhhhhhhhhhhhhh Жыл бұрын
This was not even that big of an attack. The traffic is literally less than 1 GB per second. If anything this was a skid attack which is further supported by them literally just loading one JS file over and over. This wasn't a DDOS attack, this was some kid trying out their $5 booter.
@OfoeNelson Жыл бұрын
So this is not on the same level as the ddos attack that took down google
@TheJulsMan Жыл бұрын
You got lucky because you are publicly sponsored ... but this brings up a BIGGER point. When you pay for metered services, the providers NEED to indemnify you against DDOS attacks or other potentially ruinous events. This could be a huge selling point, because not all services have built in caps. As a matter of experience, I witnessed a peer who used AWS, his application while still in beta had a memory leak and AWS sent him a $13k bill. Insane! Risk avoidance is important to any business especially if there is no ceiling or price cap. As someone who uses these services this keeps me up at night.
@Shri Жыл бұрын
It is actually really cheap to buy residential proxies (pools with millions of IP addresses) and then use them to bombard requests to services. These residential proxies exist to enable scraping of SERP content as well as regular sites with hardened DDOS protections. Some residential proxy services also bypass recaptchas (using AI and sometimes even humans) for premium. Residential proxies have legitimate use cases but can be misused to create botnets too. That is what I am suspecting is happening here. They haven't actually paid for those 600 IPs. Rather, they are tapping into a pool of million IPs provided by residential proxy services.
@NiSiRewinD Жыл бұрын
It's free, even on that scale. There are a ton of private projects for geting around state cencorship, x100.000 of IPs, since the Ukraine war shined a light on how those countries block access to Tor itself. No idea how big those requests were, but generating a couple Gbps traffic is still negible, as far as paid botnets would go. Those networks typically limit access to specific websites, but I guess it wouldn't be crazy to get around that, as user, depending on how they filter. I suspect this didn't run via Tor, but I guess we would with a list of the IPs
@lev2590 Жыл бұрын
Considering that one 10gbit server can (in theory) handle 1.5tb raw traffic (in 20min) I don't believe that this was a big DDOS attack. Also I believe that having multiple cheap vps with high bandwith automatically deployed when needed, would be probably way cheaper than vercel. Obv. the developer experience will be worse, especially when setting all the servers up or other cluster related issues occure.
@perc-ai Жыл бұрын
Yes this was just 15 yr old some kid with a very small botnet. A pro would have used 10k IPs and Vercel would have to shutdown their dns for a period of time
@dandogamer Жыл бұрын
If you weren't on the pro license and sponsored by vercel this might have been a different story. I can imagine a normal person would have to suck up the big fees or take down their application
@mormantu8561 Жыл бұрын
It depends. Cloud providers more commonly than you might think cover ridiculous fees in case of an error or attack like this. Because they don't want to lose your business.
@Knightfall23 Жыл бұрын
They’ll most likely cover it the first time but if it happens again your on the hook
@mormantu8561 Жыл бұрын
@@Knightfall23 Agreed.
@dasrite Жыл бұрын
@@Knightfall23 I'd rather get a straight response from Vercel directly than try to imagine what might happen
@samdcbu Жыл бұрын
If you are a normal user on the free tier of vercel you would just put your application behind Cloudflare for free and let them handle the DDoS traffic.
@TheLKStar Жыл бұрын
I mean, having a punchable face and arrogant personality is bound to provoke someone when exposing yourself to thousands of strangers. Even so, it takes some extra thick emotional issues to waste any amount of time and resources to get revenge on a parasocial relationship.
@unnaturalatrophy5377 Жыл бұрын
Damn dude 😂
@AJ-wf1vh Жыл бұрын
huh, so vercel has no rate limitting by default? I would have expected a managed service to handle this, not have me set up my own edge middle ware (upstash?) It doesn't look good on their part
@hanes2 Жыл бұрын
I think both CloudFlare and Linode (Akamai CDN) has ddos protection included.
@canht95 Жыл бұрын
It was probably @theprimeagen...
@alitonoliveira1700 Жыл бұрын
It would be great to know how to prevent a DDOS attack against AWS and GCP (Cloud Run and Cloud Functions).
@QckSGaming Жыл бұрын
GCP: Toggle the DDOS shield on. Cloud Armor it was called I think.
@callowaysutton Жыл бұрын
Did Vercel give more details? Such as if the IP addresses were all from the same IP block or dispersed across many, whether or not they were residential IPs, their own IPs, IPs from other cloud providers, etc, geolocation lookups of the IPs? All of this seems like it would be super useful to know about to prevent future attacks both for you and them
@bluesteelbass Жыл бұрын
Willing to bet those IP addresses got put on one of the many naughty lists that are distributed to rulesets for firewalls.
@ilijanl Жыл бұрын
Is it me, but why are they are targeting static assets? I mean if you want to increase Theo's bill, DDoS the api route which has the upstash rate limiter as well? It will cost him 0.20 cents - 0.40 cent per 100k request for upstash and probably far more for serverless/edge functions on vercel. Thus sending 100 milion request will at least cost 1000 * 0.20 + 500 GB hours ( 5*40) = 400 dollar + rest of vercel
@perc-ai Жыл бұрын
Tbh this attack probably came from a 15 yr old… this is not a serious attack tbh it’s very easy to rotate 10k residential proxies and force vercel to temporarily shutdown all ping services I could probably do it
@jocdiazm Жыл бұрын
Love that the stack you are recommending is the one that you use for your stuff. I can imagine that the people behind this were just absurdly annoyed that you are recommending tools that don't fit their certificates or what they consider is the "right move".
@Fervore_ Жыл бұрын
at what point does vercel consider the requests as a ddos attack do they use any tools? what happens if a tiny dev's app gets ddossed, would vercel refund 100% of the money by all requests that day? how long do they take to answer from the point where you're under attack to when the situation gets resolved?
@nickwoodward8195 ай бұрын
after 1 minute according to their webpage. not great really given the number of requests you could be on the hook for at that point. and no, if they don't say they'll refund your money, assume they won't.
@_va3y Жыл бұрын
>>> I think it’s quite impossible to take us down with this stack Bro, you’ve just broke the main rule of opsec 😅😅
@CoderandFilmmaker Жыл бұрын
Honestly Theo, now I am very relaxed about the decision of using T3 Stack and the services you recommend us. If even Chirp handled this insanity! Then we’re in safe hands as Solo-preneurs 😊
@TheGetawayMan Жыл бұрын
Side note - The vid I'm most looking forward to is the one you mentioned about syncing clerk with your own db 🙂
@markclynch Жыл бұрын
It's laughable how much the attacker likely spent vs what you incurred. Perfect example of mitigation, you can't stop it from happening, you have to make it too expensive for bad actors to continue.
@name_less227 Жыл бұрын
Don’t these type of attacks usually use hacked computers to help them attack?
@Hexalyse Жыл бұрын
@@name_less227 They do. It might not "cost" them anything in the literal sense. They didn't spend money most probably, if they own the botnet. BUT... usually when you own such a botnet, you can sell it, or rather rent it to people who want to conduct such attacks. So all in all, either their bought botnet time and is cost them money directly; or they used their own botnet instead of renting it, and that's money they could have earned but didn't. So the result is the same : attacking cost them money. Which I find hilarious considering the ridiculous impact they had.
@jason_v12345 Жыл бұрын
But you can stop it from happening. Rate limiting middleware can be written in a just a few lines of code.
@perc-ai Жыл бұрын
@@jason_v12345 Theo doesn’t even know what he is doing lol
@yunyang6267 Жыл бұрын
I have a question. Are those people generating these attacks going to be held accountable or there are ways to generate botnets attack and get away with it
@socialkruption Жыл бұрын
Wouldn't surprise me if some from /g/ were part of this, your videos have started making thier rounds over on the board.
@t3dotgg Жыл бұрын
Oh really? Fuck yeah finally they're gonna HATE my ass
@hyper_channel Жыл бұрын
/g/ cares about nothing but LLMs right now, highly unlikely
@varma8669 Жыл бұрын
What is /g/?
@hyper_channel Жыл бұрын
@@varma8669 4chan's technology board
@g-rexsaurus794 Жыл бұрын
LLM?
@eleldevelop-bu2md Жыл бұрын
Can someone describe the AWS scenario? What would have happened and how to do rate limiting?
@lapulapucityrider3227 Жыл бұрын
Use ec2 not serverless much better
@Daddyjs5 ай бұрын
That would be an awesome tutorial setting up ddos protection using upstash
@stevenismart Жыл бұрын
Can you do a more in depth video on how to stop DDOS and other random attacks?
@gemmaatroxxibox53229 ай бұрын
DDOS is my main concern with Serverless. With an nginx proxy you can get sub 1ms 503 responses in a DDOS and cap the number of requests per IP so that it doesn't touch your actual app code when it happens. So for a free or cheap server vs a Serverless platform, at least getting started out, I know someone can't rack up costs for me.
@Rensoku611 Жыл бұрын
6:09 I dont understand this part. Why would you fare worse if you had actual servers? They have rate limiting and IP blacklisting as well
@jacoblockwood4034 Жыл бұрын
I think he’s saying that if they were hitting the actual servers, all the endpoints would be destroyed pretty quickly
@Mitsunee_ Жыл бұрын
if you're putting the upstash ratelimiter infront of everything, how are you not hitting ratelimits on upstash?
@maddsua Жыл бұрын
Yep, serverless rocks. Won't ever go back to dedicated hardware
@user-tz6nn8iw9m Жыл бұрын
ddos a static file :genius:
@dhkatz_ Жыл бұрын
Can you not put a Vercel site behind a Cloudflare proxy?
@mike110111 Жыл бұрын
How much would it have cost? Isn’t the point they were making is that on demand computing can make your costs sky rocket?
@xavhow Жыл бұрын
Just wondering why it’s costing the attacker more? The hundreds of IPs used by the botnet are probably someone else’s.
@betterinbooks Жыл бұрын
this is a pretty cool vercel ad to be honest.
@brod515 Жыл бұрын
I can't imagine that someone decided to waste any significant ammount of money doing this. I'm wondering how they had access to 600 static ip addresses.
@udaym42046 ай бұрын
upstash rate limit is good for backend but what about fronted to save from ddos attack
@dog4ik Жыл бұрын
Must be rust foundation
@masseeerra Жыл бұрын
Would be good to get an in depth video on the specifics of how you (or the tech) delt with it 🙏
@Joseleon-ct8xz Жыл бұрын
it migth be done from the rust foundation
@sandrinjoy Жыл бұрын
we got ddosed, and vercel did nothing. 6.8TB
@benheidemann3836 Жыл бұрын
Love it, keep them coming
@rifwann Жыл бұрын
dude, im sold on this platform you are using.. what a way to advertise..
@jventura1738 Жыл бұрын
Theo makes a video about Twitch dying then gets DDOS’d hmmm… 😂 all jokes aside your services handled well!
@lightninginmyhands4878 Жыл бұрын
Where is redis in your data fetching flow?
@mike110111 Жыл бұрын
Would a simple solution be - count how many requests for each IP address, and if it goes above 100 per second you block them?
@Reveliom Жыл бұрын
your hair looks majestic dude !!!
@greendsnow Жыл бұрын
1400 GB costs USD 180 on Vercel. (First 1000 is USD 20) What're you talking about?!
@just_ppe Жыл бұрын
Nice promotional video :-)
@raiyansarker19 күн бұрын
what is the charge for the upstash service?
@Daddyjs5 ай бұрын
How would you be dead if you had actual servers running this? Wouldnt it be better because then you wont get charged a lot of money? Thanks for helping a newb like me understand
@jbbzzi Жыл бұрын
Absolutely hilarious that some people will spend thousands out of pure spite for absolutely nothing. Still, I can't help but be skeptical of these new cloud providers you're showcasing. When the 'growth' period ends and the 'taking profits' period begins, is it still going to be more attractive than AWS? We'll see!
@hseinb Жыл бұрын
How do you know the attackers have used static IP addresses?
@filda2005 Жыл бұрын
if it did not chang within 2weeks, it is not so much dynamic IP
@lescobrandon2202 Жыл бұрын
Good stuff.
@anasouardini Жыл бұрын
Instead of bringing down your services, they just gave you a topic to talk about? hilarious!!
@SeanCassiere Жыл бұрын
Does anyone have any examples of using upstash's rate-limiter with tRPC? Been using it more, and I haven't really gotten around the concept of how rate-limiting could be added to it. It most likely would be done via a middleware, but just putting up the flag for any existing repos that have it.
@liam.brewer Жыл бұрын
he covers it in his newest t3 stack course where he builds a twitter clone called chirp
@SeanCassiere Жыл бұрын
@@liam.brewer thanks! My bad for not fully checking the repo.
@SeattleSpursFan1882 Жыл бұрын
Who has the resources to pull this off? Amazon does, that's who...
@DaveTheDeveloper Жыл бұрын
How much did it actually cost you?
@pedrofelipefonsecaenunes2435 Жыл бұрын
This won me over!
@jmarbutt23 Жыл бұрын
Where is your video on the rate limiting with UpStash?
@GeorgeDicu-hs5yp Жыл бұрын
Oh nice!! Long live rate limiting.
@thejonte Жыл бұрын
Wow! Holy crap!
@JuanRodriguez-rh4kp Жыл бұрын
It was probably the angular team trying to make t3 look bad.
@nicolasguillenc Жыл бұрын
People can be nasty, that’s why it’s better sometimes to build in private. Oh and good for you man the infrastructure really held up
@marcelor1235 Жыл бұрын
What contents do you have about Vercel? I guess we all have some app to protect.
@andriiantoniuk8419 Жыл бұрын
But, what exact cost for you of this DDOS attack?
@justmrmendez Жыл бұрын
"impossible to take down our services" my guy, you are challenging the wrong community here 😂, and for those that say it cost them money, ego doesn't care about money. We do things some time just to prove that we can, no need to hate you to do something like this (I'm not saying I'm even capable of this), but if i could, i wouldn't do it because hate, but ego or passion, and seems you already have enough of both
@shahkaleem6601 Жыл бұрын
Because you were Ddosed you have got a subscriber.
@sortof3337 Жыл бұрын
What do you think about IP based rate limiters? Would they prevent such attacks? That's the only protection I have against DDOs.
@ttrss Жыл бұрын
ddos atks are (Distributed)DOS
@sortof3337 Жыл бұрын
@@ttrss so its like wearing a hat to be bulletproof. Nice.
@ttrss Жыл бұрын
@Sort of 😄i guess. And then cloudflare protection is like letting a government protect you, but they're like super authoritarian.
@rizkiaprita Жыл бұрын
i see what you did there
@Khari99 Жыл бұрын
Well you sold me on Vercel thats for sure.
@FaisalAfroz Жыл бұрын
introducing captchas might also help.
@t3dotgg Жыл бұрын
For fetching a JS file?
@spectator5144 Жыл бұрын
Vercel needs bun
@RedStone576 Жыл бұрын
bun on the edge?
@cowabunga2597 Жыл бұрын
Pork bun ?
@venicebeachsurfer Жыл бұрын
Did Vercel try to charge you?
@TomNook. Жыл бұрын
Primeagen testing out his Rust pen test code?
@foswa6335 Жыл бұрын
Theo what are these thumbnails 😂
@riolly Жыл бұрын
I thought hacker is always smart, they absolutely stupid in this case.
@Knightfall23 Жыл бұрын
Gotcha
@headlights-go-up Жыл бұрын
Considering how much this cost the attackers and how little it affected you, it had to be someone with disposable income. Seems like Elon wants to get his revenge after you told him how ads work.
@aryabp Жыл бұрын
Seems someone's back-end needs Rust Framework 😊 Edit: idk why my reply multiple times got deleted. so i am sorry, i cant explain due to no freedom of speech