How I survived a DDoS attack

[@JeffGeerling]

6:49 - It was DNS. RedShirtJeff.com

[@baylinkdashyt]

It's always DNS. Or Lupus.

[@declanmcardle]

It was vegans...

[@TheJacklwilliams]

Thanks for the post Jeff. Full of great information and I'm glad you were able to battle back against it. Per your point re centralization vs de-centralization, agreed wholeheartedly. Now, a challenge to you because well, I'm an insanely huge fan of what you do and how you do it. If you, or I, or another fan(s) were to NOT cloudfare in a case like this, what could be done to stop the attacks? The biggest issue I see that you called out, is the average home user of bandwidth is going to be doing Spectrum or AT&T, and well, pay for bandwidth... Thanks again, as always, great post, highly informative. Glad you buttoned it up.

[@GrandPlatClips]

You Leaked your IP Address 4:48

[@daveamies5031]

@@GrandPlatClips That was his previous IP before dhcp renewed 🤣🤣🤣 Pretty sure he mentioned he had a static ip in a previous episode.

[@TechnoTim]

I feel your pain Jeff! Each time I released a video talking about self-hosting security I get DDoS'd

[@Disatiere]

I can see people seeing it as a challenge

[@dieSpinnt]

@@Disatiere I can see people going to jail ...

[@Disatiere]

@@dieSpinnt I mean usually they drive there

[@dieSpinnt]

@@Disatiere Yeah, you are right. Just couldn't resist to make a pun, based on your comment:) because in reality: some of the attackers get caught ... for doing childish BS.

[@dragnar12]

U: look i have my own private server. The poeple: Lemme test how good it is

[@ghangj]

Currently on the Cyber Security road and it is amazing how much I have learned from this video, "DOCUMENT EVERYTHING".

[@RoelBaardman]

Not from security experience, but general network-admin experience: Don't just document what went wrong... also document what went right! This reveals positive patterns, shows improvements and (perhaps most important in a company) documents why the expensive tools are worth it.

[@ghangj]

@@RoelBaardman Thanks for the tip * *scribbles something* *

[@vaisakhkm783]

I also learned the same lesson a few days ago.. I never going to miss anything

[@HoloScope]

@@RoelBaardman this!

[@danielstellmon5330]

Document what went wrong for you. Document what went right for the boss.

[@paulmichals]

Thank you Jeff for this very topical video. I've been doing IT since the early 90's (yes I am older than dirt) and DOCUMENTATION (often paper notebooks) is the best bit of information to take away for those who watched this video.

[@turbopro10]

I've been doing IT since the 70s before it was called IT, so there ...

[@paulmichals]

@@turbopro10 in the 70's I was underway under water on watch as a Reactor Operator on the US Nuclear Powered fast attack submarine USS Queenfish - SSN 651. But in about '73 I do remember messing around with punch card readers at a local community college's computer lab.

[@wartlme]

Hope no one hits my site with a DDos attack. Glad you made it. Thanks for sharing.

[@volkhen0]

What’s your website? ;)

[@thrillscience]

Who would have anything against a RaspberryPi guy? Big Arduino?

[@JeffGeerling]

Heh, but Arduino's making a board with the Pico on it now. Not sure who would care that much!

[@RuiFungYip]

The nice thing about cloudflare tunnels, is that it turns an incoming connection into an outgoing connection. Which is pretty handy when you want to host a site and you're behind a CGNAT.

[@DanielLopez-up6os]

40 Mb/s Attack seems HUUUGE, Then I remembered the SpamHouse attack cloudfare protected, and it was somewhere along a 1 TB/s attack. Cloudfare is amazing AF!

[@sergsergesrgergseg]

40 mb/s is quite low.. you can buy stressers that hit a lot more than that for less than 10 dollars

[@DanielLopez-up6os]

@@sergsergesrgergseg those stressers usually are incomplete http request based tho, so quite easy to mitigate.

[@sergsergesrgergseg]

@@DanielLopez-up6os you would be surprised on the level of sophistication some of these cheaper underground services can offer

[@ernstoud]

Years ago already the adagio was that the only way to stop DDOS is making sure your pipe is bigger than theirs. There is no way around companies like Cloudflare who have the budget for those big pipes.

[@MatthewDeveloper]

This is true, I've tried blocking IP's on iptables, after a while iptables are actually using all the CPU usage on my small server. I turned the server down, waiting for the attack to be done.

[@AndrewBeeman007]

When I saw your video about the cluster on a farm I was curious as to why you didn't have it behind Cloudflare. I agree with the idea of not contributing to centralization, but there are too many bad apples out there to not have a layer of protection like Cloudflare IMO.

[@monsterhunter445]

In theory cloudflare could snoop traffic if unencrypted?

[@AndrewBeeman007]

@@monsterhunter445 If it is unencrypted, you have more significant problems. But in theory, yes.

[@webfreezy]

Just to note - you could also use AWS Cloudfront - but I don't think they have a free tier.

[@AndrewBeeman007]

@@webfreezy In my opinion, Cloudflare is far less evil than Amazon

[@soundspark]

@@AndrewBeeman007 Even though Cloudflare looks the other way at abuse?

[@vagellan_8842]

Bro! Awesome video! Love the shirt. I still plan on getting into IT professionally instead of just studying, and tinkering, and grumbling about every commercial setup I see or have problems with. Love the shirt and just bought one!

[@driver34579]

I once saw a server rack that had a glass window. There was a sign inside that read: In case of DDoS attack, break the glass and cut the cables.

[@SutherlandBoswell]

Documenting everything is the type of advice that seems obvious but is easy to skip over. I wish I had documented it, but in the past I dealt with what appeared to be a pretty small DDoS attack that turned out to actually just be a clever way a virus was trying to phone home. The domain was a simple two word name, and what I seem to remember is that both of those words happened to be in an array the virus would use to build a long list of domains to try phoning home. The malicious party could easily come back after buying a different domain from the list if they were ever shut down, and I assume it made it harder to trace back to the creator since many of the randomly generated names were already owned by legitimate sites like mine. Since my memory of it isn't great I really wish I had followed that advice, because it was an interesting learning adventure.

[@JeffGeerling]

At this point it's just my instinct-if something weird happens, immediate screenshot. If it turns out it wasn't something interesting, I can always delete the screenshot later! I've almost never had a moment where I regretted saving off some extra data during one of these moments.

[@luminescentlion]

10:53 I know the pain, it was 5Mbps up before I we switched from Comcast to Fidium now its 1Gbps up symmetrical with my down for half of what you pay.... which is nice.

[@Space_Reptile]

*adds Jeff's website to the list of websites unreachable when cloudfare has an issue again* Would love to see a project of where you make your "own cloud flare" so it won't be affected by outages like half of the Internet at this point but still be protected

[@thewhitefalcon8539]

Cloudflare can do what Cloudflare does because it has hundreds of terabits of bandwidth, and that's the only way to do it. How much do you suppose that costs?

[@Ch1spy4]

"I'm not an idiot" Red Shirt Jeff edited in "Debatable" I bet lmao

[@Wordsnwood]

Yup, that thumbnail is 🔥 (And I've tried to get my team to adopt your "it was DNS" shirt for our team uniform, but so far no go.... 😉)

[@KiwontaTv]

"How I survived a DDoS attack" - "I waited until they were done"

[@younisamedi]

God bless you Jeff! We're all with you brother.

[@airy_co]

This video is amazing, some of us like to self-host things but that comes with risks we need to be aware of!

[@MarksGoneWicked]

Several years ago, I had noticed my bandwidth taking a hit. I went into the network monitoring on my router and watched my router being slammed by requests. They were hitting the dynamic IP assigned to the router by my provider. Thankfully, only a slow connection was the only result.

[@patsypryor9850]

I just stumbled in to your site, and at my green level of understanding,I am just terrified to use my devices at all. Just unplugged my internet and went to bed. May go old school off grid,yikes!!! what an education. thnx

[@agikarasugi2294]

It’s sad that most of the DDoS traffics are from my country. A lot of people here use pirated and questionable software on both their phone and laptops without checking it first. I had my home ISP address blocked, and later found out that one of my family member’s laptop were infected with trojan and participating in DDoS.

[@maartentoors]

A "smart' device is as smart as its operator/firmware-updates, even then it can host backdoors (which can come to light using network monitoring tools). I really like your transparency/OpenSource mentality

[@cheetobambito9724]

you know what all these ddos attacks after years of never having a single one? YOURE MOVING UP IN THE TECH LIFE AND SHOULD BE PROUD!(: Good stuff Jeff cant wait for your next video

[@ianallaway4964]

Hi Jeff, I'm not sure if you're aware but the Cloudflare tunnel client (cloudflared) is actually an open source project. It would also help fix your CG-NAT problem. I've not used kubernetes but dabbled with docker swarm for a little while so I'm assuming it behave in a similar fashion. On the swarm you can run an instance of cloudflared on each node meaning (as long as every pi has an internet connection) you're no longer reliant on 1 node for the ssh tunnel. Love the channel. Thanks

[@strub3l]

While doing our charity stream to fund the British Red Cross, we had about 10k requests/second on various public services. This went on for minutes, then stopped for a few minutes, then erupted like that again. This killed the Stream for several Minutes

[@jeremygmail]

Botnets usually test their ways before they go full bore. Sometimes that is days before or in your case minutes before they go full tilt. Sorry to hear about your stream.

[@falazarte]

Amazing video! Keep up the good work.

[@rbunpat]

A question, could Cloudflare prevent this? Edit:Nevermind, I got the answer.

[@JeffGeerling]

Heh, watch to the end ;)

[@DAVIDGREGORYKERR]

I hope you have your PI Cluster behind a Pf-sense Router to stop virus attacks and other types of attack.

[@MarcoGPUtuber]

Was it DNS?

[@JeffGeerling]

Only partially :D

[@brinkoo7]

I used to fight off so many attacks like this at major hosting companies. layer 4 attacks like syn floods should generally be handled by whoever hosts the VPS... but these layer 7 attacks are a different story. I am personally not a huge fan of cloudflare for multiple reasons, but I have had pretty good success with AWS's WAF. It can be very limiting in it's rule sets and only evaluates rate limits at like 1 minute intervals... which means someone can blast you so hard in that 1 minute time period so bad you can go offline, then do it again 5 minutes later lol... the struggle is real regardless.... great video and great insights.

[@johncnorris]

Sounds like a tough day at the office but at least you've learned a lot about defensive measures.

[@luvxinh]

Interesting. I was also thinking of what you've been sharing with the world in the spirit of open source. It's basically the documentation for your infrastructure made available to the public

[@phlizneinbleedblop2318]

Woo Jeff! thanks for the info hopefully we all can prepare for inevitable attack

[@RyanHenrie999]

Maybe you could do a cyber security basics video in the future. I found this video very insightful.

[@RixtronixLAB]

Cool info, thanks for sharing, well done :)

[@CraigMullins1]

good tools to use 9:55

[@grant_vine]

You should check out what Troy Hunt has done in terms of automating adding blacklist IPs based on attacks, using Azure functions iirc but the premise is the same. Use the reporting of CF to drive adding a blacklist entry. This is probably the most basic form of Security Automation & Response (SOAR).

[@lward53]

"How I survived a DDoS attack" The DDoS wasn't big enough to knock you out then.

[@constantiusdamar1925]

Great Video Jeff,

[@marcodoe4690]

That IoT Device security part is exactly why I intend to run my devices locally. First of all I don't want to rely on different IoT Clouds which might be up to data security standards or not. And for sure I don't want to expose my devices to the internet where they could be breached and used to perform malicious attacks on other networks.

[@nopalfi1409]

Why did Indonesia have the most traffic? Does that mean that Indonesia has the most infected botnet or something?

[@pranaypallavtripathi2460]

When you are as smart as Jeff, you can make a whole video on why your viewers are not able to view your website and gain even more views. Take that DDoS attacker. 😂

[@JeffGeerling]

When life gives you lemons...

[@syntheticperson]

Very insightful. Thanks

[@fakebizPrez]

@JeffGeerling - do you recommend using separate baremetal server(s) dedicated strictly to monitoring? I have some Zimaboards and a NUC collecting dust..

[@adyanth]

This is why I stay as far away from PHP as possible. Spawning threads like crazy under load is not what I like seeing. Very well reacted under the attack I would say, and Cloudflare saves more people's behinds than I can count when stuff hits the fan :) "DOCUMENT EVERYTHING!"

[@davidgrisez]

A DDoS attack is a common problem that occurs on a number of websites. Whenever I find that it takes a long time or I can not access a website on my computer I very often suspect a DDoS attack on the website. From other videos I have seen it costs extra money to handle these DDoS attacks and get a website back online.

[@jmr]

Not CE talk! I've been playing with Cloudflare and considering it for my personal site.

[@manuelthallinger7297]

Tbh, i allways tought that using a cdn im front of a site is more or less a must have. I blocked certain countries at cdn level , 75 - 80 % of my bad traffic is from rusia, belarus and ukraine

[@wowtheworldchannel]

I like the thumbnail, It’s cool.

[@CaptZenPetabyte]

This video has been a great lesson, thanks for explaining everything Jeff! With more and more people using NAS machines, and running their own servers (not realising that what theyre doing by all those apps in their NAS's!) this is going to become more and more of a problem for everyday people, let alone those of us trying to secure HomeLabs.

[@CaptZenPetabyte]

PS. My old Asustor had a GeoBlocking App that would lock out not just IP's but the IP's of the City/State/Country ... could this be a solution for us HomeLabbers?

[@pmr1049]

When even a simple SHOUTcast server gets hit with "robots". There is nothing to gain there from a small community radio XD

[@alisaakiron]

Just set up my website on a small server in HK, 2 hours later, it got a DDoS attack. :(

[@bepowerification]

Man. i am glad you survived!!

[@kjyhh]

very imperative. good vid

[@interru_io]

Cloudflare. The biggest man in the middle attack till this day.

[@ericgeorge1797]

They probably used sapphire and a botnet

[@gokhansarioz7150]

You have nginx, you can also put a ratelimiting with a nice mapping, this way you can prevent geting POST and die, of course if transactions were millions your nginx will die but still it need good amount of money, CF is free and nice choice for sure.

[@soultracer]

Thanks for sharing.

[@DavidWilde1]

I need a shirt that says "It was the MTU"

[@nobodyimportant7655]

They use small package packs to preform definate denial of service freezing my stuff and crashing it. And I have to start over at the last save.

[@Two-Checks]

Happened to me. My solution was inverting the anti tachyon particles in the bio neural gel packs.

[@pfksr64]

Correction, Cloudflare does NOT have thousands of DDOS scrubbing pops, they have about a dozen. Now they do have thousands of POPs but most are virtual and serve assorted functions within their CDN. Because Cloudflare architects their CDS in multiple hub and spoke segments, what happens is that when you ingest into their CDN the service you are paying for will re-direct the ingress traffic to the appropriate pop. Ingest into the CDN is based on Geolocation and then it will be routed accordingly based on routing metrics to the DDOS service.

[@SchmartMaker]

Thanks for all the useful info Jeff! Can I point out one tiny thing though? The label on that Hue Bridge should spell "Philips Hue", so not "Phillips" like the screwdriver. 😉

[@JeffGeerling]

Heh, oops.

[@fedemtz6]

I would love a video on your monitoring tools

[@Cueteman]

smack filters works for me!

[@IngwiePhoenix]

Around 2010, I started a small little community with a few friends and later other people that came by. The gist of the situation was that the place I originally frequented was going through some drama because of a ban-hammer slinging, stoned to the moon admin. It was, in hindsight, quite a funny situation but rather anoying back in the day. A while after I had made the community, running with Yii 1.x and other PHP software like AJAXChat, I was hit with a multitude of DDoS attacks whilst on vacation in Egypt - and it suuuuuuuuucked. Have fun using weird pseudo-SSH clients on random websites to log into your server to look at whats up because you can't install anything on a PC in an internet cafe. And we are talking 2011, webapps - as we know them today - and things like Electron were still pipedreams. During that, I eventually ended up using CloudFlare myself but also learned, that some server providers - i used Hetzner at that time - sometimes have the ability to null-route traffic. So while I had my server recover, I had all traffic null-routed, turning my server into a black box for a few hours and accessing it later using a KVM-over-IP solution given by the DC staff. So far, CloudFlare has been my mainstay for DNS management and they have a few solid and good features. But without a good CDN, running a website off your own lonesome server can be a nightmare. Configuring rate limits per IP is one way, but usually not the best. I learned that a hardware firewall can be a figurative live-saver - because that is what Hetzner used when null-routing my traffic. So far, I have not seen other services like CloudFlare, and since I joined them they have grown tremendously. Fun fact, did you know that at some point CloudFlare itself was used as a DDoS puppet? xD Turns out that there was a mailserver bug that was abused, i think it was SMTP - and later an NTP related issue, that would allow an attacker to fake the destination address of a paket and thus utilize a gigantic network in their stead to send traffic. And CloudFlare ended up becoming the butt of that joke momentarily. Poor whoever-got-hit-by-that xD...their network is big. Hopefuly, in the future, Cloudflare, AWS and Google Cloud and Azure aren't the only platforms out there to make average websites more secure and provide good DNSes and caches. But man, did I ever giggle when I heared you describe the issue, because I FELT THAT. xD

[@SPPhotography89]

There are unlimited 4G / 5G connections in Europe, the fee is modest ~ €30 per month. The operator already filters out DDos from there.

[@yacahumax1431]

thank you for the info.

[@alexevlad]

The problem is when is not a website, and you are a provider of services and try to protect your company or customers. In my country, kiddies are ddosing services of customers mostly daily and that's a problem, what only datacenter can mitigate or try yourself, if datacenter will not block you. So what is a good way to protect the servers without cloudflare, because it's not available for IP:PORT (custom port)?

[@OldPoi77]

Who would win in a celebrity geek match? Jeff Geerling vs ExplainingComputers ???

[@devflite8782]

There's couple of reason of why Indonesia increased in becoming source of DDoS. Our gadget and device usage increased exponentially in these few years. This increase unfortunately not followed by good security practices and possibly increase the possibility of infected device with malware or botnet. Beside that, there's a recent booming of cheap Cloud VPS with hourly rate. That can be registered by any user from the world without any identification assessment. Some even priced like $0.0047/hr, and of course they have API Access. Imagine that the actor can easily deploy thousands of new instance armed for DDoS and redeploy after each hour to get fresh IP in fraction of dollars and easily joined again if got suspended.

[@wileamyp]

Another origin of the botnets is all the pirated software floating around. It's a "well-known secret" (rahasia umum) anyway.

[@SergeantPepper]

New shirt idea It was "smart" devices.

[@setheloe7090]

Bungie needs this rn bruh.

[@helgeandreasvuolab605]

I have a dream, that you make a tutorial about openmediavault 6 hardening. Firewall configuration and more.

[@KieranShort]

Oh my gosh. I know a bit about networking around the house and setting up a few lan servers.. but I wouldn't even know where to start blocking a DDoS attack. I'd probably just pull the plug.. which is very effective but, well, .... a server without power isn't a server.

[@AlanDampog]

this is a great video!!!

[@castercs]

just setup rule so it triggers after X amount simultaneous connections

[@chrisg6091]

Cloudflare Tunnel™ is also your friend. Reverse proxy for mere mortals.

[@soviut303]

You talk about cache busting but aren't responses to POST requests usually uncached?

[@Darkk6969]

For years I've been wondering about Cloudflare so I've looked into it a few months ago and made the switch. Now my personal domains are hosted and managed on Cloudflare making my life so much easier. Cloudflare e-mails me weekly reports about my domain's visibility on the internet. Also, using Cloudflare's family DNS to filter content on my home network.

[@soap1555]

if you have a paid plan at Cloudflare, you can open a support ticket and they'll provide you lots of valuable information about the attack so you can help Cloudflare block the requests in their firewall edit: it's also a great idea to add a JS challenge firewall rule for european countries that you can just enable whenever something happens

[@mrchucu1]

Have you tried POW mitigation?

[@Manguitom]

I would like to take a course on networking in order to understand most of the terms in this video. What would you recommend Jeff? I'm open to books, online courses, anything that's worth it!

[@JeffGeerling]

NetworkChuck has some decent videos going through it all.

[@fjgaston]

your website was down a few minutes ago (17:30 Paris time)

[@JeffGeerling]

Yep; looks like the DDoS cannon was fired up again this morning. Back online now, but turned on Cloudflare's 'Under Attack' mode.

[@HerrBlauzahn]

For making sure only Cloudflare accesses my site I use their Authenticated Origin Pull feature. Works like a charm and I don't have to update Firewall rules whenever their IPs change.

[@ebrocoliphoto]

seeing "Indonesia" in the list of traffic is kinda scary, the hacker in my country maybe just testing it's attack for getting ready for russia's attack

[@wileamyp]

Nope, I think it's from pirated software.

[@janklas7079]

I don't know what caching you have in place, but have you considered varnish as a frontend? also php-fpm has multiple settings which may be worth trying, like timeouts and limits to the threadpool.

[@aakarsh12]

*me hitting my own laptop based server with 10000 POST requests per second*

[@perkelatorZ79]

I Have watched nearly all of the videos you have posted and you talk a lot about documentation. Like in this video with when something happens document everything. How to you do this effective and efficiency? I understand logs, but you had a git issue on this like you are using git as a notebook of sorts. I would love to see a video on all the ways git can be used other than just for code. Currently I just use it for code. Never really thought about using the issue tracker for homelab stuff.

[@oprimeirodenorth]

3k request sounds like a DoS with proxy

[@wintrywind]

wow that traffic from indonesia, remembering internet literacy here is so low.

[@dnldnl4880]

This was excellent

[@semirauthsala6001]

Even before I click the video. I knew Cloudflare inside this somewhere lol.

[@RJ_Cormac]

Why I don't tell or show people where my stuff is when they ask about how I keep it physically and digitally safe. So many people ask these dumb questions, it seems they don't want to do a basic Google search on solutions to their problem, or have malicious intentions.

[@devluz]

Scary. This stuff is what keeps me away from cloud services. In this situation they probably scale up until I file bankruptcy :s

[@swunbox]

Average day in my life. But instead of pis I use and AMD EPYC as a host system, DDoS protection and cloudflare proxy. And fun fact: ARM CPUs can handle more requests per second than x86 CPUs. So for a proxy, a VPS with and ARM CPU and nginx can handle more requests

[@chrisakaschulbus4903]

I know this would be pretty much impossible to measure or calculate... but i would love to know how much electric energy is wasted on ddos. Not only the bots doing requests, but the servers trying to deal with it.

[@peerpede-p.]

DDOS/DNS attacks is one of the oldest trick in the book, mid Mars the Swedish bank identification system went down half a day, 'technical problems they stated'... Wells someone with a lot of machines in control, stated this DDos attack.