How I survived a DDoS attack

[@JeffGeerling]

6:49 - It was DNS. RedShirtJeff.com

[@baylinkdashyt]

It's always DNS. Or Lupus.

[@declanmcardle]

It was vegans...

[@TheJacklwilliams]

Thanks for the post Jeff. Full of great information and I'm glad you were able to battle back against it. Per your point re centralization vs de-centralization, agreed wholeheartedly. Now, a challenge to you because well, I'm an insanely huge fan of what you do and how you do it. If you, or I, or another fan(s) were to NOT cloudfare in a case like this, what could be done to stop the attacks? The biggest issue I see that you called out, is the average home user of bandwidth is going to be doing Spectrum or AT&T, and well, pay for bandwidth... Thanks again, as always, great post, highly informative. Glad you buttoned it up.

[@GrandPlatClips]

You Leaked your IP Address 4:48

[@daveamies5031]

@@GrandPlatClips That was his previous IP before dhcp renewed 🤣🤣🤣 Pretty sure he mentioned he had a static ip in a previous episode.

[@ghangj]

Currently on the Cyber Security road and it is amazing how much I have learned from this video, "DOCUMENT EVERYTHING".

[@RoelBaardman]

Not from security experience, but general network-admin experience: Don't just document what went wrong... also document what went right! This reveals positive patterns, shows improvements and (perhaps most important in a company) documents why the expensive tools are worth it.

[@ghangj]

@@RoelBaardman Thanks for the tip * *scribbles something* *

[@vaisakhkm783]

I also learned the same lesson a few days ago.. I never going to miss anything

[@HoloScope]

@@RoelBaardman this!

[@danielstellmon5330]

Document what went wrong for you. Document what went right for the boss.

[@marcogenovesi8570]

the big question is why would anybody waste all those resources to ddos a content creator personal website. Tell us the truth, you have been working with secret stuff and these are the other secret agents coming at you

[@JeffGeerling]

I'm going to have to ask Red Shirt Jeff what shenanigans he's been up to...

[@kalam564]

@@JeffGeerling We've traced the DDoS and it's coming from inside the house. (insert suspenseful music)

[@JTSabre]

Probably an attack against the VPS hosts range of IPs, rather than a direct attack on the single site.

[@PBRichfield]

@@JasonWade plausible false flag attack ttp

[@JWSmythe]

It's most likely not a government. It's some script kiddie with his botnet, trying to impress his friends. That happens a lot, most people just don't see many of them. For botnets, it isn't a lot of "resources". The resources are the infected or exploited machines scattered all over the world. If he had done more logging and analysis, he could have formed a good idea of how many separate attackers there were. If he were less public and less accountable, he could have looked at the attacking machines. They may have all had the same kind of remote exploit. There are a lot that even use things like Word Press to relay their attacks. I have been called to help fix exactly that. Someone used some unpatched exploit, to deploy a bunch of back doors and relay code, and attacks were being run from their site. I really hate Word Press just because there are so many holes. Every WP site is just waiting for a script kiddies crawler to discover, and to add to their botnet.

[@izzieb]

Some people are sad and have too much time. Why would anyone DDoS Jeff's site?!

[@cybergaming42424]

Well at least it led to content

[@jeremygmail]

Why? It is the Internet. Things happen for the dumbest reasons like because I can or because I am doing it for the lolz. The only thing you can do is protect yourself as much as you can and watch/alert on suspicious anomalies. That is more than just high traffic these days so your alerting has to be on point.

[@ApolloSevan]

Red shirt Jeff egged them on I’m sure! 🤣

[@luis449bp]

Just for fun

[@JeffGeerling]

@@jeremygmail most likely someone did it for the lulz. They're probably chuckling to themselves watching this video :P

[@TechnoTim]

I feel your pain Jeff! Each time I released a video talking about self-hosting security I get DDoS'd

[@Disatiere]

I can see people seeing it as a challenge

[@dieSpinnt]

@@Disatiere I can see people going to jail ...

[@Disatiere]

@@dieSpinnt I mean usually they drive there

[@dieSpinnt]

@@Disatiere Yeah, you are right. Just couldn't resist to make a pun, based on your comment:) because in reality: some of the attackers get caught ... for doing childish BS.

[@dragnar12]

U: look i have my own private server. The poeple: Lemme test how good it is

[@jeremygmail]

Thank you for sharing @Jeff. As someone who was on an IT team which dealt with a Top 3 (at the time) attack where they sustained 300Gb/s and wreaked havoc on the network. I can relate. Good luck keeping ahead of the botnets!

[@halo4life166]

Out of curiosity, was this an OVH one? I seem to remember a Blackhat talk about something similar in the past

[@jeremygmail]

@@halo4life166 no it was a different one. Just saw they got to 1Tbps. That is pretty egregious for 2016!

[@jesseclutterbuck6617]

these days its done with time servers. you can request alot of data from a time server and we all know how fast there connection speeds are. this technique is called the amplified ddos attack. in the past some amplified ddos attacks have reached as high as 2.5 tbps.

[@rickharold7884]

Wow. That’s nuts. Great learning experience and love that u share it. Much appreciated

[@jimo8486]

lol use ovh

[@paulmichals]

Thank you Jeff for this very topical video. I've been doing IT since the early 90's (yes I am older than dirt) and DOCUMENTATION (often paper notebooks) is the best bit of information to take away for those who watched this video.

[@turbopro10]

I've been doing IT since the 70s before it was called IT, so there ...

[@paulmichals]

@@turbopro10 in the 70's I was underway under water on watch as a Reactor Operator on the US Nuclear Powered fast attack submarine USS Queenfish - SSN 651. But in about '73 I do remember messing around with punch card readers at a local community college's computer lab.

[@hubertnnn]

One suggestion to make this kind of DDoS less problematic: Use two servers. One that will be handling heavily cachable data for most people and another one for handling POSTs etc. This way if you get DDoSed, only the vulnerable POST server will get hit, and the GET server will survive pretty much unharmed. You will loose the ability to send comments, but the website will still be up and fine.

[@logangraham2956]

you could still send comments , just accept posts only from your post server and ignore everything else on your get server. than when i comment is posted client -> post server -> get server ->client

[@novianindy887]

Do most ddos use post requests? do the Get requests only have minimum impact on the server?

[@hubertnnn]

@@novianindy887 No, but GET requests can be cached and POST cannot. Introducing a caching layer can increase the number of requests per second 100-1000 times. So the same server will be able to handle just 100 POST requests per second or 10.000 GET requests per second. It will be much harder to DDOS the second one. Also most traffic in websites is GET traffic, so sacrificing a POST server to DDOS will only limit functionality instead of killing the website completely.

[@Quint2105]

@@hubertnnnJust curious. I’m running a VPs which got hit by small scale ddos attacks recently. Hereby they targeted the vps ip itself. The problem we faced is that our network bandwidth ran out, our system resources such as CPU and RAM where almost unaffected. Our normal legitimate traffic could not get trough the network bombardment of the attack anymore. What could I do to prevent this from happening apart from switching to a higher bandwidth network?

[@hubertnnn]

@@Quint2105 It really depends on the specifics of your system. Not sure what do you mean by VP. But as generic rules I would start with a CDN network with DDOS protection like cloudflare (it has free tiers). Next thing would be to reduce the size of responses, if they filled your network then a lot of data had to be transferred. After that I would add some kind of IP based throttling (typical configuration is 60 reqests per minute per IP), it wont help against huge botnet, but will at least limit the effects and help against smaller botnets (limiting each bot to just 1 req/s). And yes, increasing the available bandwith could also help. You could also try auth protecting some of the larger data behind short lived tokens that require authentication and captcha, though capthas recently can be easily solved by machines while being hard to solve by humans..

[@DevOdyssey]

Awesome breakdown Jeff! This really affirms everything I've learned in my job so far! I've certainly learned about DDoS, but never seen such an eloquently "documented" video, describing your real life, personal experience in a timeline manner. Happy Cloudflare came in and saved the day. And of course, the obligatory, "Thanks DNS, thanks" 😒

[@almostmatt1tas]

This is one of those videos that makes me realise I don't know as much about computing as I thought I did. Time to spend the day googling acronyms! Thanks for sharing your experience Jeff.

[@CraigEngbrecht]

Thanks! I appreciate all your arguments here, and the wonderful breakdown of the information. I have always argued for monitoring, however, rarely implimented it correctly. :P

[@zade69420]

Woah, I've never seen anyone use the thanks feature before.

[@douglasbubbletrousers4763]

This is blowing my mind.

[@bitonic589]

someone translate to $usd

[@livepdfan4708]

@@bitonic589 ~$4 (3.98) USD

[@pendragonscode]

I had this happen to me once before! Instead of doing what you did, I decided to make my site give a link to youtube... A rickroll link. After that, I added captcha and then my site got up. Meanwhile for some bizzare reason, the rickroll thing i added actually attracted more attention as my friends who knew about my site started sharing it around lol. I didn't know my site was down until the 2nd day, added the rickroll on the 2nd day, left it like that for almost a week. (Was also at the same time getting a new machine to host the thing.). So yes rickroll helped me.

[@LifeIsRecusive]

"I got hacked, because i revealed my infrastructure" *Makes additional video exposing more of the infrastructure information* Red shirt jeff should have done this video lol Thanks for the explainer, always appreciated

[@trbry.]

I always thought **hacked** was more of 'you locked that door not this' instead of "I'm gonna put all these billions on pebbles on the road so you can't drive here".

[@Max24871]

At this point he just using us to test his hardening efforts

[@levelup1279]

I have a broad definition of hacking, & that's just manipulating computer systems in a way which the designer never intended, or just doing general hacker things. It's hard to define hacking because of how broad it is. The 90's were much more liberal with who qualified as a "hacker". Now there are all these keyboard warriors who get angry if you don't use the correct terminology. "That's not hacking you idiot, its exploiting". Nope, it's all hacking.

[@Mmmm_tea]

@@trbry. some people line their drives with pebbles,they don't stop you driving just stop you driving fast... if you like your windows.

[@bjw8qsrmhgxn4wwk30]

Security by obscurity is a farce. With some sleuthing you’d be able to determine almost any information about Jeff’s site.

[@xuldevelopers]

I see you can use pretty nifty awk and friends. The usual thing I do in these cases is to use it to select, say, all IPs that requested more than 10 pages without ever downloading any other resources (CSS, JS, images whatever is required for your pages). That always works because these DDoS do not simulate browsers completely so it is easy to differentiate robots. One must deal with legit SE later. Then I feed the firewall. Once it runs again there is a plenty of time to do other things. The biggest list of IPs I selected that way was 35.000 individual IPs during one attack. Also there are those tell signs, that you can target with grep|cut|awk|sort|uniq|... most of DDoS attacks rotate UA strings that get logged in your logs. So selecting and grouping all requests from given IP and seeing how many different UA strings compared to requests is there turned out to be very often a reliable way. With other signs it is close to 100% accuracy. Not to mention that if all it does is to hit POST page then it is easy to identify all "weird" IPs. Worst cases are those DDoS attacks that simulate a normal browsing. I've seen behavior where there were sequences of dozen or so pages that each robot followed pretending to be a normal user. That was a tough one because server was under pressure and everywhere I looked all appeared to be a legit user browsing until you figure out that all you see is the same browsing sequence over and over. I added logging of select cookies and some HTTP headers like supported languages and such so I have more info to use for selection. Those robots very rarely support cookies. Especially tracking JS-set cookies is something attackers don't support. But lately I noticed that few attackers were stuffing standard cookies like tracking cookies or session cookies with random numbers but it is really rare.

[@JeffGeerling]

This is true, but at one point today DigitalOcean shut down my main IP after it was getting hit with 2.3 million PPS, at which point I basically hid the entire server behind Cloudflare. Some providers may be willing to work with DDoS mitigation, but usually once it's persistent and high volume, they want that traffic out of their DC. For the initial attacks, I could've handled it by setting up fail2ban with Nginx logs, but once it got going, I fear I would've needed to invoke CF regardless :(

[@Supremax67]

@@JeffGeerling -- Also a reason why decentralized public ledger are trending.

[@francois1e4]

@@Supremax67 What do you mean?

[@Supremax67]

@@francois1e4 -- You call them blockchain, but that is over simplifying it. Not every public DLT is a ledger and not every blockchain is actually decentralized. In a sea of noise, only a few of them shows promise. The next decade should be interesting.

[@francois1e4]

@@Supremax67 True that!

[@skug978]

Thanks for sharing Jeff. Good that you dealt with the problem and gained the experience from it.

[@john_hawley]

So from what I'm gathering from your analysis: For the home guy with limited bandwidth and hardware your options are: 1. Buy a PaaS (i.e. Cloudflare) 2. Shut 'er down

[@JeffGeerling]

Pretty much. Though if you are close friends with a local ISP, you might be able to work with them on a solution. But good luck with that if you're 99% of people. Spectrum won't give me the time of day :(

[@abhimaanmayadam5713]

Cloudflare does have a free tier

[@jeremygmail]

@@JeffGeerling Ha! when we got ddos'ed our provider took us offline because the botnet was killing their network too :)

[@Vangard21]

I'm no crypto-advocate (and ATM it's like 90% scams), but distributed Web 3 is an alternative to Cloudflare/AWS/Google control of the internet. IPFS for statics, Ethereum dapp backend. And a ~2 minute page load for end users :/ But it might well turn out to be the best alternative to web centralization.

[@Xamy-]

@@Vangard21 no mate. That shit is all just a ploy to promote crypto scams, don’t talk about it. Watch “Line goes up - the problem with NFTs” (and crypto)

[@filovirus1]

whoever comes up with a way to pinpoint DDoS attackers so we can reach out and slap their physical faces should win a Nobel prize

[@johndododoe1411]

More likely an award from a military team, such as Nobel's original corporations.

[@chrisakaschulbus4903]

YES! Then darknet users can finally find out who is ddosing their markets. That'd be great.

[@angryjoshi]

Child assault is illegal 😂

[@glynnetolar4423]

A little "wet work" might curb that kind of activity.

[@TheNillquest]

Please, more videos about prevention of ddos and ransomware, btw brazilian here, sorry for our country been a part of the attack IT security here is minimal.

[@JeffGeerling]

I don't blame individuals ;) Some people like to block entire countries-and that can help to an extent-but I would rather leave things as open as possible because even in the countries where it seems the worst of these attacks originate (especially Russia and a few south Asian countries), there are still plenty of legitimate users who just want to learn something, and who am I to shut them off?

[@Private-GtngxNMBKvYzXyPq]

Glasnost -> Peace Cooperation -> Mutual Benefit I second the request for more videos on security. Thank you.

[@thrillscience]

Who would have anything against a RaspberryPi guy? Big Arduino?

[@JeffGeerling]

Heh, but Arduino's making a board with the Pico on it now. Not sure who would care that much!

[@microm4n]

This was great. I was debating putting my website behind Cloudflare in preparation for an attack that I can't cope with myself, along with some of their other offerings (like the new anti phishing email stuff). I too am not a fan of the centralisation of traffic but for now it's about the only option we have, and CF are still "good guys", at least for now.

[@wartlme]

Hope no one hits my site with a DDos attack. Glad you made it. Thanks for sharing.

[@volkhen0]

What’s your website? ;)

[@davidbubble6863]

Curious how those attackers choose their targets. Jeff's web site of all things? Makes no sense at all.

[@JeffGeerling]

I once learned from a wise old man... "Some people just want to watch the world burn."

[@davidbubble6863]

Well that's one reason 😂

[@guiorgy]

@@davidbubble6863 Just for the LOLs because YOLO? Or they happen to be a viewer who wanted to challenge Jeff, or give him a reason to make this video ¯\_(ツ)_/¯

[@RetroGameStream]

Yeah I wonder that same thing. I host over 200 websites and the few times I've had to deal with this they were always the smaller sites that didn't make any sense, like a ma pa grocery store or small church. Not sure what they got out of that unless they just chose their sites randomly.

[@AudreyRobinel]

@@RetroGameStream perhaps they are just trying their tools, see what works or not? maybe they are akin to "interns" in their fields, and this is their assignment before leveling up?

[@roguethinker6284]

Smokin Video Geoff. In 11 minutes you've covered just about everything I know about mitigating DDOS attacks. Took me years. My brain is getting old

[@eyesofnova]

I don't know much about it, but I've run across the github repo for Gatekeeper. Its open source DDOS protection. I'd be curious how well it functions in practice, or how hard it is to get it configured correctly.

[@lorenzo42p]

probably not a fix for a ddos. the best you can do is drop the packets, but the flood of packets still needs to reach the firewall before they can be dropped. bottleneck is your internet connection, which gets swamped and overloaded. there are some possible options to drop the packets before they're sent to your internet connection, but those technologies are usually reserved for the big companies.

[@johndododoe1411]

@@lorenzo42p Yeah, I wish there was a common ICMP extension for a swamped server to request upstream dropping of high volumes of packets. Something that could be quietly running on the Cisco backed routers and prioritize blocking requests that reject the most attack traffic in any given moment, letting through low bandwidth traffic that happens to hit site firewall rules that send too many block requests. Ideally the router priority software would also detect if multiple recipients are requesting protection against the same outside source, ultimately resulting in zombified machines getting blocked closer to their own connections, following by an angry letter from their ISP.

[@MarcoGPUtuber]

0:54 It's a good thing you use CRTs. The lack of smart features make them UNHACKABLE!

[@qingdom]

"Anton died so that we could live!" - Gilfoyle, Pied Piper

[@RuiFungYip]

The nice thing about cloudflare tunnels, is that it turns an incoming connection into an outgoing connection. Which is pretty handy when you want to host a site and you're behind a CGNAT.

[@hse5.0]

Nice documenting the attack. Also looking forward to the GPU project for the pi. Looks like someone deserves a well rest this weekend 😜.

[@scottwilliams895]

Jeff, it's very cool of you to share what happened, how you responded, and what you learned. Content like this is why you earned my Sub years ago, and why I keep coming back for more.

[@AndrewBeeman007]

When I saw your video about the cluster on a farm I was curious as to why you didn't have it behind Cloudflare. I agree with the idea of not contributing to centralization, but there are too many bad apples out there to not have a layer of protection like Cloudflare IMO.

[@monsterhunter445]

In theory cloudflare could snoop traffic if unencrypted?

[@AndrewBeeman007]

@@monsterhunter445 If it is unencrypted, you have more significant problems. But in theory, yes.

[@webfreezy]

Just to note - you could also use AWS Cloudfront - but I don't think they have a free tier.

[@AndrewBeeman007]

@@webfreezy In my opinion, Cloudflare is far less evil than Amazon

[@soundspark]

@@AndrewBeeman007 Even though Cloudflare looks the other way at abuse?

[@DanielLopez-up6os]

40 Mb/s Attack seems HUUUGE, Then I remembered the SpamHouse attack cloudfare protected, and it was somewhere along a 1 TB/s attack. Cloudfare is amazing AF!

[@sergsergesrgergseg]

40 mb/s is quite low.. you can buy stressers that hit a lot more than that for less than 10 dollars

[@DanielLopez-up6os]

@@sergsergesrgergseg those stressers usually are incomplete http request based tho, so quite easy to mitigate.

[@sergsergesrgergseg]

@@DanielLopez-up6os you would be surprised on the level of sophistication some of these cheaper underground services can offer

[@ernstoud]

Years ago already the adagio was that the only way to stop DDOS is making sure your pipe is bigger than theirs. There is no way around companies like Cloudflare who have the budget for those big pipes.

[@MatthewDeveloper]

This is true, I've tried blocking IP's on iptables, after a while iptables are actually using all the CPU usage on my small server. I turned the server down, waiting for the attack to be done.

[@Ch1spy4]

"I'm not an idiot" Red Shirt Jeff edited in "Debatable" I bet lmao

[@janhumpolicek8373]

Holy cow you saved me! I amexperiencing this rn!!!! Thank so so much.

[@RicardoVargas03]

Man! You are AMAZING! This is the first time I have seen your videos; there is a LOT of value here! Thank you!

[@adversHandle]

I accidentally dosed thr online learning portal for my college once . The webpage wasn't loading I left the tab open and did other work. 2 Hours later the admin knocked on the door of the study room asking if I was in there 🥺😱 what. He was cool about it I had no idea I took down the website 😅

[@JeffGeerling]

Haha, though that shouldn't be on you, probably an application bug that caused your browser to keep reloading something in an infinite redirect loop or something!

[@driver34579]

I once saw a server rack that had a glass window. There was a sign inside that read: In case of DDoS attack, break the glass and cut the cables.

[@AndrewDanne]

Good to hear you are back on line and in 1 piece after this. Can you suggest how I would test/monitor my IoT, Raspberry PI's, Network, to see & monitor if I am contributing to a BOT net? Cheers

[@vagellan_8842]

Bro! Awesome video! Love the shirt. I still plan on getting into IT professionally instead of just studying, and tinkering, and grumbling about every commercial setup I see or have problems with. Love the shirt and just bought one!

[@sebastiannielsen]

Note that using a firewall (instead of Cloudflare which he uses in this video) doesn't work if you have a limited line to your ISP. If the strength of the DDoS attack is bigger than your incoming internet line, only person in charge to stop the DDoS is your ISP or upstream hosting provider. This because even if you have a imaginary, perfect, firewall that is able to absorb 100% of the DDoS attack and let 100% of legitimate traffic in (which doesn't exist in reality), your internet line would still be swamped with the DDoS attack, which means the filtering must happen before the bandwidth is reduced. Another reason mitigations must be upstream, is if you have a so called metered connection. Even if your firewall blocks the traffic, it will usually still count against the metering, why you need to talk to hosting provider regardless. As saw in the video, he is using Cloudflare, which acts as a big firewall before it even reaches your hosting provider, thus your smaller internet line isn't affected. This is equvalient with mitigating at your hosting providers' backbone. Smaller DDoS attacks however, can be mitigated with a good anti-DDoS protection to not load down the server.

[@ewookiis]

All lines / connections are limited ;). Cloudflare and services as such does have firewalls, but the descision is not always made at the lowest level at first on these kind of services. The saving grace is the blocking (fw's) of known bad, loadbalancing, caching and the much higher ceiling of bandwidth since they have a multitude of ingress points - also the known flows of sender and destination across cloudflare setup accumulate quite a nice dataflow, in conjunction with known addresses from botnets etc etc. in short - one always needs an backup ;).

[@sebastiannielsen]

@@appxprt4648 Yes 50% of total capacity, since the system wont be able to respond. But usually, broadband is metered in like 100mbit/100mbit, so a DDoS attack has to fill either of these to 100%, which is equal to 50% total. Backplane capacite is usually number of ports / 2, so a 16 port gbit switch usually have 8gbit backplane, so you would just not be able to flood it unless you have access to multiple ports on that switch. Or have access to a unfiltered uplink port. But these types of DDoS attacks can be mitigated by a firewall, ergo, make sure there is a filter before uplink port. Its when the DDoS are bigger than your ISP connection that you are in trouble.

[@maartentoors]

I love the transparency of your content/tutorials. As for monitoring I use NEMS myself (awesome package).As for mitigating the 3rd attackon your site, 30 mins response/mitigation... KUDOS! Cheers from a fan!

[@SutherlandBoswell]

Documenting everything is the type of advice that seems obvious but is easy to skip over. I wish I had documented it, but in the past I dealt with what appeared to be a pretty small DDoS attack that turned out to actually just be a clever way a virus was trying to phone home. The domain was a simple two word name, and what I seem to remember is that both of those words happened to be in an array the virus would use to build a long list of domains to try phoning home. The malicious party could easily come back after buying a different domain from the list if they were ever shut down, and I assume it made it harder to trace back to the creator since many of the randomly generated names were already owned by legitimate sites like mine. Since my memory of it isn't great I really wish I had followed that advice, because it was an interesting learning adventure.

[@JeffGeerling]

At this point it's just my instinct-if something weird happens, immediate screenshot. If it turns out it wasn't something interesting, I can always delete the screenshot later! I've almost never had a moment where I regretted saving off some extra data during one of these moments.

[@arekx]

Previous video was talking only about frontend nginx on VPS - doing caching etc. So what was php-fpm doing on that VPS and eating resources in ddos case? It was supposed to be nginx only doing cache and pushing things to backend on raspberry pi cluster. So no php needed at all on vps.

[@JeffGeerling]

Indeed it was, but quickly after the DDoS started, I moved the database back to the VPS and tried transitioning Drupal's traffic to it. To make that happen quickly, I had set up Nginx on that server to still direct _some_ types of requests to the main VPS instead of back at the Pi cluster. Honestly, I should probably take a deeper look at the logs though, because I am also surprised so many requests were hitting the VPS's PHP handler while the rest were hitting the backend. I wonder if it could've been related to an http vs https configuration error on my end.

[@muhammadazmi3323]

yep I can confirm this, most people in my country don't care about cybersecurity even on a government level, no wonder how many botnets have already been installed on individual devices

[@karter61]

I run quite a popular website that gets multiple daily massive DDoS attack attempts. Cloudflare is a godsend as without it there is no way the site would be able to stay up. I have got quite a bit of complicated rules running on CF to help prevent these attacks. The best thing is that CF has really great API's so I have been able to automate everything to keep the site online

[@xephael3485]

This is basically an advertisement for cloudflare... you didn't handle anything. Also 3000 requests per second? That's pretty weak bro.... 2kpps is minimum alarming level for most DDoS mitigation products.

[@techbriefing]

yeah 3k RPS is very low and if your site collapses at that level of traffic it's a bit embarrassing most modern DDoS attacks on medium to large services are now 2-3M RPS+

[@techbriefing]

the largest ever DDoS attack was performed by someone I know, who owns the Meris botnet. that achieved 400 million RPS by exploring a vulnerability in HTTP/2 (now known as the Rapid Reset vulnerability). he has previously taken the entire Cloudflare network offline, taken Google offline, taken Amazon offline, among other huge services. he's been thwarted by Cloudflare and Google teaming up but he's already found a new vulnerability although I don't know the details.

[@KiwontaTv]

"How I survived a DDoS attack" - "I waited until they were done"

[@memesfrdayz9932]

bro said hundreds of countrys

[@jessequartey]

People didn't understand you.

[@luminescentlion]

10:53 I know the pain, it was 5Mbps up before I we switched from Comcast to Fidium now its 1Gbps up symmetrical with my down for half of what you pay.... which is nice.

[@alexlandherr]

Sad to hear that, I run my own little Dark Web site hosting satellite images on a Pi4B 8GB using Nginx. I hope I never experience this. EDIT 2022-03-16 19:46 UTC: It’s a static site so not that much going on, it’s meant to be lightweight. No JS, only CSS for styling using Atomic.

[@them2545]

Oh cool mind dropping the onion link

[@skorpion1298]

@@them2545 I like onions

[@Space_Reptile]

*adds Jeff's website to the list of websites unreachable when cloudfare has an issue again* Would love to see a project of where you make your "own cloud flare" so it won't be affected by outages like half of the Internet at this point but still be protected

[@thewhitefalcon8539]

Cloudflare can do what Cloudflare does because it has hundreds of terabits of bandwidth, and that's the only way to do it. How much do you suppose that costs?

[@ur1friend437]

OMG Jeff I hope the attacks didn't took too much time from your family time. I admire your work and honesty, so for that reason is heartbreaking to see you being punish, for your good work and honesty.

[@superbrain3848]

reminds me on the Mirai Botnet that managed to shut down a quite large part of the internet back in 2016. Some kids managed to create a massive botned in an accident, and then launched an Attack to the wrong IP, causing the DNS provider Dyn to run into issues. Dyn provides a DNS service for websites like Spotify, soundcloud and Twitter.

[@rbunpat]

A question, could Cloudflare prevent this? Edit:Nevermind, I got the answer.

[@JeffGeerling]

Heh, watch to the end ;)

[@MarksGoneWicked]

Several years ago, I had noticed my bandwidth taking a hit. I went into the network monitoring on my router and watched my router being slammed by requests. They were hitting the dynamic IP assigned to the router by my provider. Thankfully, only a slow connection was the only result.

[@Alok_raj]

Mine also got ddos Thks it might help me.

[@Jason-mk3nn]

Great video, on all levels! Great work and thank you for sharing!

[@MarcoGPUtuber]

Was it DNS?

[@JeffGeerling]

Only partially :D

[@meddlin]

Good work, man! This is awesome, and inspires me to beef up my website more.

[@pranaypallavtripathi2460]

When you are as smart as Jeff, you can make a whole video on why your viewers are not able to view your website and gain even more views. Take that DDoS attacker. 😂

[@JeffGeerling]

When life gives you lemons...

[@reggiep75]

I have to say I laughed at '..maybe I shouldn't have tempted fate..' part. You just know people would've thought 'Yeah, we gotta see if we can beat this website into submission!'.

[@michaeldesilets7528]

I enjoy your videos for entertainment. When I let my head get out of entertainment mode and back into semi work mode I learn a bit and enjoy your videos more. Thank you.

[@JeffGeerling]

Heh, when worlds collide!

[@jackfletch2001]

1. Identify 2. Probable Cause 3. Test 4. Implement 5. Verify 6. Document A+ drilled these steps into my head.

[@FelipeFonsecaRocha]

Nice from you to not stop sharing man... Really learn a lot from you...

[@Wordsnwood]

Yup, that thumbnail is 🔥 (And I've tried to get my team to adopt your "it was DNS" shirt for our team uniform, but so far no go.... 😉)

[@tdragon87]

I have it set up like this asweel, block everything but cloudflare. You could also set up rate limit rules on Cloudflare, but you have to be pretty relaxed with that, especially if, like you said, drupal has some weird ways of doing things. Also you could try something like High availability proxy, and once the server get's bombarded by a DDOS attack it would reroute traffic to a server with much stricter rules. This last part is just something I thought of now and am not sure how it would pan out.

[@FlygisTheFlygis]

“How I survived a sneeze attack” up next on this channel. So glad you’re alive bro

[@younisamedi]

God bless you Jeff! We're all with you brother.

[@RobertFabiano]

This was a great video! Real meaty subject with good level of detail

[@kewitt1]

My home lab has 11 subdomain, I've never noticed a ddos attack because my websites can all do a api call to my firewall and shutdown individual ip from getting to my reverse proxy server. If to much traffic inbound traffic it limits the ip, if that ip continues requests it bans for 1 hour then 24 then firewall redirect all web requests to a removal requests form which requires email code. Which you have access to for 24 hours then blocked forever. Currently my firewall has 52121

[@agikarasugi2294]

It’s sad that most of the DDoS traffics are from my country. A lot of people here use pirated and questionable software on both their phone and laptops without checking it first. I had my home ISP address blocked, and later found out that one of my family member’s laptop were infected with trojan and participating in DDoS.

[@henkdevries5042]

Holy!! You too?? Wondering how much sleep you have missed, it did not show. Good work Jeff, keep up the great work!

[@airy_co]

This video is amazing, some of us like to self-host things but that comes with risks we need to be aware of!

[@Abishek_Muthian]

Only issue with CloudFlare is that they're very good at what they do that we become quickly reliant upon them for everything, Leading to all eggs in one basket. We need more CloudFlare like services. Also touching upon the learning, Doing away with PHP by choosing a concurrency native language like Go can help build distributed web-services without 3rd party frameworks incl. a web-server like ngnix. Not to mention the massive performance gains by ditching python and thereby saving hosting costs.

[@linuxastro]

Yep, it has gotten bad of late. I had a "what's that noise" event when an alarm went off for the first time last week (10 years after install).

[@hongkonghacker]

If it is a flood DDoS, you can't blocked it by your own, it is attack your circuit.

[@povilasstaniulis9484]

Thank you for sharing. Documenting as much information as possible is an incident response 101 for pretty much everyone who is hosting their own servers. Of course, not everyone runs high-profile websites/projects to require very extensive monitoring, but some level of monitoring is pretty much a must. And monitoring is useful not just for cyber incidents but for monitoring overall server health too. A good example of a tool many admins don't bother to set up is root emails. They aren't that hard to set up and a simple email from smartctl that you hard disk is going south can prevent data loss and downtime. Or an email telling that your backup script didn't run properly.

[@JeffGeerling]

For me it's usually the once or twice per year certbot starts complaining about certs... I then fix it before the cert expires :D

[@falazarte]

Amazing video! Keep up the good work.

[@martinc.7424]

Thank you for sharing your experience Jeff.

[@LoftechUK]

It’s always good to have the experience of this if it doesn’t effect your pocket.

[@ricardomarques748]

Hey Jeff! You could try to use Crowdsec

[@ricardomarques748]

@@rainerwahnsinn3265 right!? I installed it on my cluster and did some test. It seems to be working… he should totally try it and the attacker should attack him again lol

[@strub3l]

While doing our charity stream to fund the British Red Cross, we had about 10k requests/second on various public services. This went on for minutes, then stopped for a few minutes, then erupted like that again. This killed the Stream for several Minutes

[@jeremygmail]

Botnets usually test their ways before they go full bore. Sometimes that is days before or in your case minutes before they go full tilt. Sorry to hear about your stream.

[@lward53]

"How I survived a DDoS attack" The DDoS wasn't big enough to knock you out then.

[@carstenr.1682]

Thank you for sharing your insights on this. I'm using the Cloudflare Argo Tunnel. It's super easy and the big benefit is, that you do not need to open port 80 and 443 for inbound traffic. I can also recommend to use an AWS Lightsail instances - they are cheap, fast ...

[@Star-xf8rd]

For Network monitoring I can recommend PRRG it’s quite powerful with lots of sensors prebuilt, you can even create own sensors for numerous systems

[@DerekPeldo]

Are you going to post a list of the attacking IP's? I'd love to compare that with my honeypot stats or even the list Russia posted of ddos attackers hitting them.

[@JeffGeerling]

Check the issue here-only printed the first few dozen though github.com/geerlingguy/jeffgeerling-com/issues/141

[@DerekPeldo]

@@JeffGeerling I see a 2 of the ip's that hit you also checked out my honeypot over the last few months. I'd still love to see a more complete list of attackers if you wouldn't mind (and have the time of course).

[@JeffGeerling]

@@DerekPeldo Here are more to check: github.com/geerlingguy/jeffgeerling-com/issues/141#issuecomment-1069479879

[@DerekPeldo]

@@JeffGeerling Thanks Jeff! I don't have any of those IP's hitting my honeypot. I'm going to throw your lists together and monitor for traffic to/from those IP's for a few days to curate a block list. It's interesting that a handful of those new IP's are cloudflare IP's.

[@Rosco785]

Loved this deep dive into this, bell and subbed for sure.

[@DAVIDGREGORYKERR]

I hope you have your PI Cluster behind a Pf-sense Router to stop virus attacks and other types of attack.

[@patsypryor9850]

I just stumbled in to your site, and at my green level of understanding,I am just terrified to use my devices at all. Just unplugged my internet and went to bed. May go old school off grid,yikes!!! what an education. thnx

[@user-xw6fg5pi8q]

I have been in this field for a while now, and my website is being viewed by thousands of users daily. As a result, I got excellent Cloudflare rules to prevent bots from attacking any of my website content and rate-limiting on Nginx and access.log monitoring on my backend to enable custom Cloudflare rules. While doing all of this using a free plan on Cloudflare, I can safely say it is an end-game. I can block Million requests per second without much worry, and if my CPU goes high enough, the captcha is always here to stop everything.

[@ianallaway4964]

Hi Jeff, I'm not sure if you're aware but the Cloudflare tunnel client (cloudflared) is actually an open source project. It would also help fix your CG-NAT problem. I've not used kubernetes but dabbled with docker swarm for a little while so I'm assuming it behave in a similar fashion. On the swarm you can run an instance of cloudflared on each node meaning (as long as every pi has an internet connection) you're no longer reliant on 1 node for the ssh tunnel. Love the channel. Thanks

[@ShinyTechThings]

That's nuts, I put together some scripts that help with stuff like this, I'll probably make a video on it sometime for NGINX.

[@Daryan997]

I personally faced this problem on a game server and my horrible VPS provider did not provide any DDOS protection, first thing i did was to switch to a different VPS provider but something i've learned is to never open port 80 on a VPS server unless you host a website on it but later i removed it and put it on a hosting. Second time i faced this problem was on my website and there was a horrible attack, honestly there wasn't a perfect solution but cloudflare did help me

[@jimbo-dev]

Oh interesting, I’m considering similar setup, but my idea was to build my own anycast cdn on vultr. Vultr because it’s the only provider which supports bringing your own ip address space so in theory I could expand the service to other service providers

[@RixtronixLAB]

Cool info, thanks for sharing, well done :)

[@maartentoors]

A "smart' device is as smart as its operator/firmware-updates, even then it can host backdoors (which can come to light using network monitoring tools). I really like your transparency/OpenSource mentality

[@n0madfernan257]

It was the power of PI's that prevented it, believe me

[@Girz0r]

Eye opening. May redesign BTC node plans, as well as the Doom2 multiplayer build :)

[@marcuskobel6562]

Brazil is in the list of attackers. What a shame. 🤦🏻‍♂️