Thanks for the post Jeff. Full of great information and I'm glad you were able to battle back against it. Per your point re centralization vs de-centralization, agreed wholeheartedly. Now, a challenge to you because well, I'm an insanely huge fan of what you do and how you do it. If you, or I, or another fan(s) were to NOT cloudfare in a case like this, what could be done to stop the attacks? The biggest issue I see that you called out, is the average home user of bandwidth is going to be doing Spectrum or AT&T, and well, pay for bandwidth... Thanks again, as always, great post, highly informative. Glad you buttoned it up.
@GrandPlatClips2 жыл бұрын
You Leaked your IP Address 4:48
@daveamies50312 жыл бұрын
@@GrandPlatClips That was his previous IP before dhcp renewed 🤣🤣🤣 Pretty sure he mentioned he had a static ip in a previous episode.
@ghangj2 жыл бұрын
Currently on the Cyber Security road and it is amazing how much I have learned from this video, "DOCUMENT EVERYTHING".
@RoelBaardman2 жыл бұрын
Not from security experience, but general network-admin experience: Don't just document what went wrong... also document what went right! This reveals positive patterns, shows improvements and (perhaps most important in a company) documents why the expensive tools are worth it.
@ghangj2 жыл бұрын
@@RoelBaardman Thanks for the tip * *scribbles something* *
@vaisakh_km2 жыл бұрын
I also learned the same lesson a few days ago.. I never going to miss anything
@HoloScope2 жыл бұрын
@@RoelBaardman this!
@danielstellmon53302 жыл бұрын
Document what went wrong for you. Document what went right for the boss.
@marcogenovesi85702 жыл бұрын
the big question is why would anybody waste all those resources to ddos a content creator personal website. Tell us the truth, you have been working with secret stuff and these are the other secret agents coming at you
@JeffGeerling2 жыл бұрын
I'm going to have to ask Red Shirt Jeff what shenanigans he's been up to...
@kalam5642 жыл бұрын
@@JeffGeerling We've traced the DDoS and it's coming from inside the house. (insert suspenseful music)
@JTSabre2 жыл бұрын
Probably an attack against the VPS hosts range of IPs, rather than a direct attack on the single site.
@PBRichfield2 жыл бұрын
@@JasonWade plausible false flag attack ttp
@JWSmythe2 жыл бұрын
It's most likely not a government. It's some script kiddie with his botnet, trying to impress his friends. That happens a lot, most people just don't see many of them. For botnets, it isn't a lot of "resources". The resources are the infected or exploited machines scattered all over the world. If he had done more logging and analysis, he could have formed a good idea of how many separate attackers there were. If he were less public and less accountable, he could have looked at the attacking machines. They may have all had the same kind of remote exploit. There are a lot that even use things like Word Press to relay their attacks. I have been called to help fix exactly that. Someone used some unpatched exploit, to deploy a bunch of back doors and relay code, and attacks were being run from their site. I really hate Word Press just because there are so many holes. Every WP site is just waiting for a script kiddies crawler to discover, and to add to their botnet.
@izzieb2 жыл бұрын
Some people are sad and have too much time. Why would anyone DDoS Jeff's site?!
@cybergaming424242 жыл бұрын
Well at least it led to content
@jeremygmail2 жыл бұрын
Why? It is the Internet. Things happen for the dumbest reasons like because I can or because I am doing it for the lolz. The only thing you can do is protect yourself as much as you can and watch/alert on suspicious anomalies. That is more than just high traffic these days so your alerting has to be on point.
@ApolloSevan2 жыл бұрын
Red shirt Jeff egged them on I’m sure! 🤣
@luis449bp2 жыл бұрын
Just for fun
@JeffGeerling2 жыл бұрын
@@jeremygmail most likely someone did it for the lulz. They're probably chuckling to themselves watching this video :P
@TechnoTim2 жыл бұрын
I feel your pain Jeff! Each time I released a video talking about self-hosting security I get DDoS'd
@Disatiere2 жыл бұрын
I can see people seeing it as a challenge
@dieSpinnt2 жыл бұрын
@@Disatiere I can see people going to jail ...
@Disatiere2 жыл бұрын
@@dieSpinnt I mean usually they drive there
@dieSpinnt2 жыл бұрын
@@Disatiere Yeah, you are right. Just couldn't resist to make a pun, based on your comment:) because in reality: some of the attackers get caught ... for doing childish BS.
@dragnar12 Жыл бұрын
U: look i have my own private server. The poeple: Lemme test how good it is
@hubertnnn2 жыл бұрын
One suggestion to make this kind of DDoS less problematic: Use two servers. One that will be handling heavily cachable data for most people and another one for handling POSTs etc. This way if you get DDoSed, only the vulnerable POST server will get hit, and the GET server will survive pretty much unharmed. You will loose the ability to send comments, but the website will still be up and fine.
@logangraham29562 жыл бұрын
you could still send comments , just accept posts only from your post server and ignore everything else on your get server. than when i comment is posted client -> post server -> get server ->client
@novianindy887 Жыл бұрын
Do most ddos use post requests? do the Get requests only have minimum impact on the server?
@hubertnnn Жыл бұрын
@@novianindy887 No, but GET requests can be cached and POST cannot. Introducing a caching layer can increase the number of requests per second 100-1000 times. So the same server will be able to handle just 100 POST requests per second or 10.000 GET requests per second. It will be much harder to DDOS the second one. Also most traffic in websites is GET traffic, so sacrificing a POST server to DDOS will only limit functionality instead of killing the website completely.
@Quint2105 Жыл бұрын
@@hubertnnnJust curious. I’m running a VPs which got hit by small scale ddos attacks recently. Hereby they targeted the vps ip itself. The problem we faced is that our network bandwidth ran out, our system resources such as CPU and RAM where almost unaffected. Our normal legitimate traffic could not get trough the network bombardment of the attack anymore. What could I do to prevent this from happening apart from switching to a higher bandwidth network?
@hubertnnn Жыл бұрын
@@Quint2105 It really depends on the specifics of your system. Not sure what do you mean by VP. But as generic rules I would start with a CDN network with DDOS protection like cloudflare (it has free tiers). Next thing would be to reduce the size of responses, if they filled your network then a lot of data had to be transferred. After that I would add some kind of IP based throttling (typical configuration is 60 reqests per minute per IP), it wont help against huge botnet, but will at least limit the effects and help against smaller botnets (limiting each bot to just 1 req/s). And yes, increasing the available bandwith could also help. You could also try auth protecting some of the larger data behind short lived tokens that require authentication and captcha, though capthas recently can be easily solved by machines while being hard to solve by humans..
@paulmichals2 жыл бұрын
Thank you Jeff for this very topical video. I've been doing IT since the early 90's (yes I am older than dirt) and DOCUMENTATION (often paper notebooks) is the best bit of information to take away for those who watched this video.
@turbopro102 жыл бұрын
I've been doing IT since the 70s before it was called IT, so there ...
@paulmichals2 жыл бұрын
@@turbopro10 in the 70's I was underway under water on watch as a Reactor Operator on the US Nuclear Powered fast attack submarine USS Queenfish - SSN 651. But in about '73 I do remember messing around with punch card readers at a local community college's computer lab.
@rickharold78842 жыл бұрын
Wow. That’s nuts. Great learning experience and love that u share it. Much appreciated
@jimo84862 жыл бұрын
lol use ovh
@pendragonscode2 жыл бұрын
I had this happen to me once before! Instead of doing what you did, I decided to make my site give a link to youtube... A rickroll link. After that, I added captcha and then my site got up. Meanwhile for some bizzare reason, the rickroll thing i added actually attracted more attention as my friends who knew about my site started sharing it around lol. I didn't know my site was down until the 2nd day, added the rickroll on the 2nd day, left it like that for almost a week. (Was also at the same time getting a new machine to host the thing.). So yes rickroll helped me.
@DevOdyssey2 жыл бұрын
Awesome breakdown Jeff! This really affirms everything I've learned in my job so far! I've certainly learned about DDoS, but never seen such an eloquently "documented" video, describing your real life, personal experience in a timeline manner. Happy Cloudflare came in and saved the day. And of course, the obligatory, "Thanks DNS, thanks" 😒
@almostmatt1tas2 жыл бұрын
This is one of those videos that makes me realise I don't know as much about computing as I thought I did. Time to spend the day googling acronyms! Thanks for sharing your experience Jeff.
@CraigEngbrecht2 жыл бұрын
Thanks! I appreciate all your arguments here, and the wonderful breakdown of the information. I have always argued for monitoring, however, rarely implimented it correctly. :P
@zade694202 жыл бұрын
Woah, I've never seen anyone use the thanks feature before.
@douglasbubbletrousers47632 жыл бұрын
This is blowing my mind.
@bitonic5892 жыл бұрын
someone translate to $usd
@livepdfan47082 жыл бұрын
@@bitonic589 ~$4 (3.98) USD
@xuldevelopers2 жыл бұрын
I see you can use pretty nifty awk and friends. The usual thing I do in these cases is to use it to select, say, all IPs that requested more than 10 pages without ever downloading any other resources (CSS, JS, images whatever is required for your pages). That always works because these DDoS do not simulate browsers completely so it is easy to differentiate robots. One must deal with legit SE later. Then I feed the firewall. Once it runs again there is a plenty of time to do other things. The biggest list of IPs I selected that way was 35.000 individual IPs during one attack. Also there are those tell signs, that you can target with grep|cut|awk|sort|uniq|... most of DDoS attacks rotate UA strings that get logged in your logs. So selecting and grouping all requests from given IP and seeing how many different UA strings compared to requests is there turned out to be very often a reliable way. With other signs it is close to 100% accuracy. Not to mention that if all it does is to hit POST page then it is easy to identify all "weird" IPs. Worst cases are those DDoS attacks that simulate a normal browsing. I've seen behavior where there were sequences of dozen or so pages that each robot followed pretending to be a normal user. That was a tough one because server was under pressure and everywhere I looked all appeared to be a legit user browsing until you figure out that all you see is the same browsing sequence over and over. I added logging of select cookies and some HTTP headers like supported languages and such so I have more info to use for selection. Those robots very rarely support cookies. Especially tracking JS-set cookies is something attackers don't support. But lately I noticed that few attackers were stuffing standard cookies like tracking cookies or session cookies with random numbers but it is really rare.
@JeffGeerling2 жыл бұрын
This is true, but at one point today DigitalOcean shut down my main IP after it was getting hit with 2.3 million PPS, at which point I basically hid the entire server behind Cloudflare. Some providers may be willing to work with DDoS mitigation, but usually once it's persistent and high volume, they want that traffic out of their DC. For the initial attacks, I could've handled it by setting up fail2ban with Nginx logs, but once it got going, I fear I would've needed to invoke CF regardless :(
@Supremax672 жыл бұрын
@@JeffGeerling -- Also a reason why decentralized public ledger are trending.
@francois1e42 жыл бұрын
@@Supremax67 What do you mean?
@Supremax672 жыл бұрын
@@francois1e4 -- You call them blockchain, but that is over simplifying it. Not every public DLT is a ledger and not every blockchain is actually decentralized. In a sea of noise, only a few of them shows promise. The next decade should be interesting.
@francois1e42 жыл бұрын
@@Supremax67 True that!
@skug9782 жыл бұрын
Thanks for sharing Jeff. Good that you dealt with the problem and gained the experience from it.
@LifeIsRecusive2 жыл бұрын
"I got hacked, because i revealed my infrastructure" *Makes additional video exposing more of the infrastructure information* Red shirt jeff should have done this video lol Thanks for the explainer, always appreciated
@trbry.2 жыл бұрын
I always thought **hacked** was more of 'you locked that door not this' instead of "I'm gonna put all these billions on pebbles on the road so you can't drive here".
@Max248712 жыл бұрын
At this point he just using us to test his hardening efforts
@levelup12792 жыл бұрын
I have a broad definition of hacking, & that's just manipulating computer systems in a way which the designer never intended, or just doing general hacker things. It's hard to define hacking because of how broad it is. The 90's were much more liberal with who qualified as a "hacker". Now there are all these keyboard warriors who get angry if you don't use the correct terminology. "That's not hacking you idiot, its exploiting". Nope, it's all hacking.
@Mmmm_tea2 жыл бұрын
@@trbry. some people line their drives with pebbles,they don't stop you driving just stop you driving fast... if you like your windows.
@bjw8qsrmhgxn4wwk302 жыл бұрын
Security by obscurity is a farce. With some sleuthing you’d be able to determine almost any information about Jeff’s site.
@TheNillquest2 жыл бұрын
Please, more videos about prevention of ddos and ransomware, btw brazilian here, sorry for our country been a part of the attack IT security here is minimal.
@JeffGeerling2 жыл бұрын
I don't blame individuals ;) Some people like to block entire countries-and that can help to an extent-but I would rather leave things as open as possible because even in the countries where it seems the worst of these attacks originate (especially Russia and a few south Asian countries), there are still plenty of legitimate users who just want to learn something, and who am I to shut them off?
@Private-GtngxNMBKvYzXyPq2 жыл бұрын
Glasnost -> Peace Cooperation -> Mutual Benefit I second the request for more videos on security. Thank you.
@scottwilliams8952 жыл бұрын
Jeff, it's very cool of you to share what happened, how you responded, and what you learned. Content like this is why you earned my Sub years ago, and why I keep coming back for more.
@roguethinker62842 жыл бұрын
Smokin Video Geoff. In 11 minutes you've covered just about everything I know about mitigating DDOS attacks. Took me years. My brain is getting old
@filovirus12 жыл бұрын
whoever comes up with a way to pinpoint DDoS attackers so we can reach out and slap their physical faces should win a Nobel prize
@johndododoe14112 жыл бұрын
More likely an award from a military team, such as Nobel's original corporations.
@chrisakaschulbus4903 Жыл бұрын
YES! Then darknet users can finally find out who is ddosing their markets. That'd be great.
@angryjoshi Жыл бұрын
Child assault is illegal 😂
@glynnetolar442311 ай бұрын
A little "wet work" might curb that kind of activity.
@hse5.02 жыл бұрын
Nice documenting the attack. Also looking forward to the GPU project for the pi. Looks like someone deserves a well rest this weekend 😜.
@microm4n2 жыл бұрын
This was great. I was debating putting my website behind Cloudflare in preparation for an attack that I can't cope with myself, along with some of their other offerings (like the new anti phishing email stuff). I too am not a fan of the centralisation of traffic but for now it's about the only option we have, and CF are still "good guys", at least for now.
@wartlme2 жыл бұрын
Hope no one hits my site with a DDos attack. Glad you made it. Thanks for sharing.
@volkhen02 жыл бұрын
What’s your website? ;)
@RicardoVargas032 жыл бұрын
Man! You are AMAZING! This is the first time I have seen your videos; there is a LOT of value here! Thank you!
@KiwontaTv10 ай бұрын
"How I survived a DDoS attack" - "I waited until they were done"
@vagellan_88422 жыл бұрын
Bro! Awesome video! Love the shirt. I still plan on getting into IT professionally instead of just studying, and tinkering, and grumbling about every commercial setup I see or have problems with. Love the shirt and just bought one!
@MarcoGPUtuber2 жыл бұрын
0:54 It's a good thing you use CRTs. The lack of smart features make them UNHACKABLE!
@RuiFungYip2 жыл бұрын
The nice thing about cloudflare tunnels, is that it turns an incoming connection into an outgoing connection. Which is pretty handy when you want to host a site and you're behind a CGNAT.
@thrillscience2 жыл бұрын
Who would have anything against a RaspberryPi guy? Big Arduino?
@JeffGeerling2 жыл бұрын
Heh, but Arduino's making a board with the Pico on it now. Not sure who would care that much!
@FelipeFonsecaRocha2 жыл бұрын
Nice from you to not stop sharing man... Really learn a lot from you...
@AndrewDanne2 жыл бұрын
Good to hear you are back on line and in 1 piece after this. Can you suggest how I would test/monitor my IoT, Raspberry PI's, Network, to see & monitor if I am contributing to a BOT net? Cheers
@maartentoors2 жыл бұрын
I love the transparency of your content/tutorials. As for monitoring I use NEMS myself (awesome package).As for mitigating the 3rd attackon your site, 30 mins response/mitigation... KUDOS! Cheers from a fan!
@DanielLopez-up6os2 жыл бұрын
40 Mb/s Attack seems HUUUGE, Then I remembered the SpamHouse attack cloudfare protected, and it was somewhere along a 1 TB/s attack. Cloudfare is amazing AF!
@sergsergesrgergseg2 жыл бұрын
40 mb/s is quite low.. you can buy stressers that hit a lot more than that for less than 10 dollars
@DanielLopez-up6os2 жыл бұрын
@@sergsergesrgergseg those stressers usually are incomplete http request based tho, so quite easy to mitigate.
@sergsergesrgergseg2 жыл бұрын
@@DanielLopez-up6os you would be surprised on the level of sophistication some of these cheaper underground services can offer
@bagorolin2 жыл бұрын
Thanks for sharing!!❤️
@luminescentlion11 ай бұрын
10:53 I know the pain, it was 5Mbps up before I we switched from Comcast to Fidium now its 1Gbps up symmetrical with my down for half of what you pay.... which is nice.
@AndrewBeeman0072 жыл бұрын
When I saw your video about the cluster on a farm I was curious as to why you didn't have it behind Cloudflare. I agree with the idea of not contributing to centralization, but there are too many bad apples out there to not have a layer of protection like Cloudflare IMO.
@monsterhunter4452 жыл бұрын
In theory cloudflare could snoop traffic if unencrypted?
@AndrewBeeman0072 жыл бұрын
@@monsterhunter445 If it is unencrypted, you have more significant problems. But in theory, yes.
@webfreezy2 жыл бұрын
Just to note - you could also use AWS Cloudfront - but I don't think they have a free tier.
@AndrewBeeman0072 жыл бұрын
@@webfreezy In my opinion, Cloudflare is far less evil than Amazon
@soundspark2 жыл бұрын
@@AndrewBeeman007 Even though Cloudflare looks the other way at abuse?
@davidbubble68632 жыл бұрын
Curious how those attackers choose their targets. Jeff's web site of all things? Makes no sense at all.
@JeffGeerling2 жыл бұрын
I once learned from a wise old man... "Some people just want to watch the world burn."
@davidbubble68632 жыл бұрын
Well that's one reason 😂
@guiorgy2 жыл бұрын
@@davidbubble6863 Just for the LOLs because YOLO? Or they happen to be a viewer who wanted to challenge Jeff, or give him a reason to make this video ¯\_(ツ)_/¯
@RetroGameStream2 жыл бұрын
Yeah I wonder that same thing. I host over 200 websites and the few times I've had to deal with this they were always the smaller sites that didn't make any sense, like a ma pa grocery store or small church. Not sure what they got out of that unless they just chose their sites randomly.
@AudreyRobinel2 жыл бұрын
@@RetroGameStream perhaps they are just trying their tools, see what works or not? maybe they are akin to "interns" in their fields, and this is their assignment before leveling up?
@ernstoud2 жыл бұрын
Years ago already the adagio was that the only way to stop DDOS is making sure your pipe is bigger than theirs. There is no way around companies like Cloudflare who have the budget for those big pipes.
@MatthewDeveloper2 жыл бұрын
This is true, I've tried blocking IP's on iptables, after a while iptables are actually using all the CPU usage on my small server. I turned the server down, waiting for the attack to be done.
@michaeldesilets75282 жыл бұрын
I enjoy your videos for entertainment. When I let my head get out of entertainment mode and back into semi work mode I learn a bit and enjoy your videos more. Thank you.
@JeffGeerling2 жыл бұрын
Heh, when worlds collide!
@muhammadazmi33232 жыл бұрын
yep I can confirm this, most people in my country don't care about cybersecurity even on a government level, no wonder how many botnets have already been installed on individual devices
@Space_Reptile2 жыл бұрын
*adds Jeff's website to the list of websites unreachable when cloudfare has an issue again* Would love to see a project of where you make your "own cloud flare" so it won't be affected by outages like half of the Internet at this point but still be protected
@thewhitefalcon85392 жыл бұрын
Cloudflare can do what Cloudflare does because it has hundreds of terabits of bandwidth, and that's the only way to do it. How much do you suppose that costs?
@john_hawley2 жыл бұрын
So from what I'm gathering from your analysis: For the home guy with limited bandwidth and hardware your options are: 1. Buy a PaaS (i.e. Cloudflare) 2. Shut 'er down
@JeffGeerling2 жыл бұрын
Pretty much. Though if you are close friends with a local ISP, you might be able to work with them on a solution. But good luck with that if you're 99% of people. Spectrum won't give me the time of day :(
@abhimaanmayadam57132 жыл бұрын
Cloudflare does have a free tier
@jeremygmail2 жыл бұрын
@@JeffGeerling Ha! when we got ddos'ed our provider took us offline because the botnet was killing their network too :)
@Vangard212 жыл бұрын
I'm no crypto-advocate (and ATM it's like 90% scams), but distributed Web 3 is an alternative to Cloudflare/AWS/Google control of the internet. IPFS for statics, Ethereum dapp backend. And a ~2 minute page load for end users :/ But it might well turn out to be the best alternative to web centralization.
@Xamy-2 жыл бұрын
@@Vangard21 no mate. That shit is all just a ploy to promote crypto scams, don’t talk about it. Watch “Line goes up - the problem with NFTs” (and crypto)
@RixtronixLAB Жыл бұрын
Cool info, thanks for sharing, well done :)
@eyesofnova2 жыл бұрын
I don't know much about it, but I've run across the github repo for Gatekeeper. Its open source DDOS protection. I'd be curious how well it functions in practice, or how hard it is to get it configured correctly.
@lorenzo42p2 жыл бұрын
probably not a fix for a ddos. the best you can do is drop the packets, but the flood of packets still needs to reach the firewall before they can be dropped. bottleneck is your internet connection, which gets swamped and overloaded. there are some possible options to drop the packets before they're sent to your internet connection, but those technologies are usually reserved for the big companies.
@johndododoe14112 жыл бұрын
@@lorenzo42p Yeah, I wish there was a common ICMP extension for a swamped server to request upstream dropping of high volumes of packets. Something that could be quietly running on the Cisco backed routers and prioritize blocking requests that reject the most attack traffic in any given moment, letting through low bandwidth traffic that happens to hit site firewall rules that send too many block requests. Ideally the router priority software would also detect if multiple recipients are requesting protection against the same outside source, ultimately resulting in zombified machines getting blocked closer to their own connections, following by an angry letter from their ISP.
@Jason-mk3nn2 жыл бұрын
Great video, on all levels! Great work and thank you for sharing!
@SutherlandBoswell2 жыл бұрын
Documenting everything is the type of advice that seems obvious but is easy to skip over. I wish I had documented it, but in the past I dealt with what appeared to be a pretty small DDoS attack that turned out to actually just be a clever way a virus was trying to phone home. The domain was a simple two word name, and what I seem to remember is that both of those words happened to be in an array the virus would use to build a long list of domains to try phoning home. The malicious party could easily come back after buying a different domain from the list if they were ever shut down, and I assume it made it harder to trace back to the creator since many of the randomly generated names were already owned by legitimate sites like mine. Since my memory of it isn't great I really wish I had followed that advice, because it was an interesting learning adventure.
@JeffGeerling2 жыл бұрын
At this point it's just my instinct-if something weird happens, immediate screenshot. If it turns out it wasn't something interesting, I can always delete the screenshot later! I've almost never had a moment where I regretted saving off some extra data during one of these moments.
@meddlin2 жыл бұрын
Good work, man! This is awesome, and inspires me to beef up my website more.
@sebastiannielsen2 жыл бұрын
Note that using a firewall (instead of Cloudflare which he uses in this video) doesn't work if you have a limited line to your ISP. If the strength of the DDoS attack is bigger than your incoming internet line, only person in charge to stop the DDoS is your ISP or upstream hosting provider. This because even if you have a imaginary, perfect, firewall that is able to absorb 100% of the DDoS attack and let 100% of legitimate traffic in (which doesn't exist in reality), your internet line would still be swamped with the DDoS attack, which means the filtering must happen before the bandwidth is reduced. Another reason mitigations must be upstream, is if you have a so called metered connection. Even if your firewall blocks the traffic, it will usually still count against the metering, why you need to talk to hosting provider regardless. As saw in the video, he is using Cloudflare, which acts as a big firewall before it even reaches your hosting provider, thus your smaller internet line isn't affected. This is equvalient with mitigating at your hosting providers' backbone. Smaller DDoS attacks however, can be mitigated with a good anti-DDoS protection to not load down the server.
@ewookiis2 жыл бұрын
All lines / connections are limited ;). Cloudflare and services as such does have firewalls, but the descision is not always made at the lowest level at first on these kind of services. The saving grace is the blocking (fw's) of known bad, loadbalancing, caching and the much higher ceiling of bandwidth since they have a multitude of ingress points - also the known flows of sender and destination across cloudflare setup accumulate quite a nice dataflow, in conjunction with known addresses from botnets etc etc. in short - one always needs an backup ;).
@sebastiannielsen11 ай бұрын
@@appxprt4648 Yes 50% of total capacity, since the system wont be able to respond. But usually, broadband is metered in like 100mbit/100mbit, so a DDoS attack has to fill either of these to 100%, which is equal to 50% total. Backplane capacite is usually number of ports / 2, so a 16 port gbit switch usually have 8gbit backplane, so you would just not be able to flood it unless you have access to multiple ports on that switch. Or have access to a unfiltered uplink port. But these types of DDoS attacks can be mitigated by a firewall, ergo, make sure there is a filter before uplink port. Its when the DDoS are bigger than your ISP connection that you are in trouble.
@karter612 жыл бұрын
I run quite a popular website that gets multiple daily massive DDoS attack attempts. Cloudflare is a godsend as without it there is no way the site would be able to stay up. I have got quite a bit of complicated rules running on CF to help prevent these attacks. The best thing is that CF has really great API's so I have been able to automate everything to keep the site online
@janhumpolicek83732 жыл бұрын
Holy cow you saved me! I amexperiencing this rn!!!! Thank so so much.
@superbrain38482 жыл бұрын
reminds me on the Mirai Botnet that managed to shut down a quite large part of the internet back in 2016. Some kids managed to create a massive botned in an accident, and then launched an Attack to the wrong IP, causing the DNS provider Dyn to run into issues. Dyn provides a DNS service for websites like Spotify, soundcloud and Twitter.
@qingdom2 жыл бұрын
"Anton died so that we could live!" - Gilfoyle, Pied Piper
@povilasstaniulis94842 жыл бұрын
Thank you for sharing. Documenting as much information as possible is an incident response 101 for pretty much everyone who is hosting their own servers. Of course, not everyone runs high-profile websites/projects to require very extensive monitoring, but some level of monitoring is pretty much a must. And monitoring is useful not just for cyber incidents but for monitoring overall server health too. A good example of a tool many admins don't bother to set up is root emails. They aren't that hard to set up and a simple email from smartctl that you hard disk is going south can prevent data loss and downtime. Or an email telling that your backup script didn't run properly.
@JeffGeerling2 жыл бұрын
For me it's usually the once or twice per year certbot starts complaining about certs... I then fix it before the cert expires :D
@adversHandle2 жыл бұрын
I accidentally dosed thr online learning portal for my college once . The webpage wasn't loading I left the tab open and did other work. 2 Hours later the admin knocked on the door of the study room asking if I was in there 🥺😱 what. He was cool about it I had no idea I took down the website 😅
@JeffGeerling2 жыл бұрын
Haha, though that shouldn't be on you, probably an application bug that caused your browser to keep reloading something in an infinite redirect loop or something!
@RobertFabiano2 жыл бұрын
This was a great video! Real meaty subject with good level of detail
@ur1friend4372 жыл бұрын
OMG Jeff I hope the attacks didn't took too much time from your family time. I admire your work and honesty, so for that reason is heartbreaking to see you being punish, for your good work and honesty.
@Rosco7852 жыл бұрын
Loved this deep dive into this, bell and subbed for sure.
@alexlandherr2 жыл бұрын
Sad to hear that, I run my own little Dark Web site hosting satellite images on a Pi4B 8GB using Nginx. I hope I never experience this. EDIT 2022-03-16 19:46 UTC: It’s a static site so not that much going on, it’s meant to be lightweight. No JS, only CSS for styling using Atomic.
@them25452 жыл бұрын
Oh cool mind dropping the onion link
@skorpion12982 жыл бұрын
@@them2545 I like onions
@fonte9352 жыл бұрын
Fascinating! Thanks Jeff.
@Alok_raj2 жыл бұрын
Mine also got ddos Thks it might help me.
@henkdevries50422 жыл бұрын
Holy!! You too?? Wondering how much sleep you have missed, it did not show. Good work Jeff, keep up the great work!
@rbunpat2 жыл бұрын
A question, could Cloudflare prevent this? Edit:Nevermind, I got the answer.
@JeffGeerling2 жыл бұрын
Heh, watch to the end ;)
@falazarte2 жыл бұрын
Amazing video! Keep up the good work.
@memesfrdayz99322 жыл бұрын
bro said hundreds of countrys
@jessequartey4 ай бұрын
People didn't understand you.
@unserfa2 жыл бұрын
Thank you for sharing!
@xephael34852 жыл бұрын
This is basically an advertisement for cloudflare... you didn't handle anything. Also 3000 requests per second? That's pretty weak bro.... 2kpps is minimum alarming level for most DDoS mitigation products.
@techbriefing Жыл бұрын
yeah 3k RPS is very low and if your site collapses at that level of traffic it's a bit embarrassing most modern DDoS attacks on medium to large services are now 2-3M RPS+
@techbriefing Жыл бұрын
the largest ever DDoS attack was performed by someone I know, who owns the Meris botnet. that achieved 400 million RPS by exploring a vulnerability in HTTP/2 (now known as the Rapid Reset vulnerability). he has previously taken the entire Cloudflare network offline, taken Google offline, taken Amazon offline, among other huge services. he's been thwarted by Cloudflare and Google teaming up but he's already found a new vulnerability although I don't know the details.
@driver34579 Жыл бұрын
I once saw a server rack that had a glass window. There was a sign inside that read: In case of DDoS attack, break the glass and cut the cables.
@MarcoGPUtuber2 жыл бұрын
Was it DNS?
@JeffGeerling2 жыл бұрын
Only partially :D
@syntheticperson2 жыл бұрын
Very insightful. Thanks
@pranaypallavtripathi24602 жыл бұрын
When you are as smart as Jeff, you can make a whole video on why your viewers are not able to view your website and gain even more views. Take that DDoS attacker. 😂
@JeffGeerling2 жыл бұрын
When life gives you lemons...
@martinc.74242 жыл бұрын
Thank you for sharing your experience Jeff.
@0x0-y3q2 жыл бұрын
Another great video thanks Jeff
@constantiusdamar19252 жыл бұрын
Great Video Jeff,
@younisamedi2 жыл бұрын
God bless you Jeff! We're all with you brother.
@soultracer2 жыл бұрын
Thanks for sharing.
@MarksGoneWicked2 жыл бұрын
Several years ago, I had noticed my bandwidth taking a hit. I went into the network monitoring on my router and watched my router being slammed by requests. They were hitting the dynamic IP assigned to the router by my provider. Thankfully, only a slow connection was the only result.
@tdragon8711 ай бұрын
I have it set up like this asweel, block everything but cloudflare. You could also set up rate limit rules on Cloudflare, but you have to be pretty relaxed with that, especially if, like you said, drupal has some weird ways of doing things. Also you could try something like High availability proxy, and once the server get's bombarded by a DDOS attack it would reroute traffic to a server with much stricter rules. This last part is just something I thought of now and am not sure how it would pan out.
@Wordsnwood2 жыл бұрын
Yup, that thumbnail is 🔥 (And I've tried to get my team to adopt your "it was DNS" shirt for our team uniform, but so far no go.... 😉)
@airy_co2 жыл бұрын
This video is amazing, some of us like to self-host things but that comes with risks we need to be aware of!
@patsypryor98502 жыл бұрын
I just stumbled in to your site, and at my green level of understanding,I am just terrified to use my devices at all. Just unplugged my internet and went to bed. May go old school off grid,yikes!!! what an education. thnx
@linuxastro2 жыл бұрын
Yep, it has gotten bad of late. I had a "what's that noise" event when an alarm went off for the first time last week (10 years after install).
@carstenr.16822 жыл бұрын
Thank you for sharing your insights on this. I'm using the Cloudflare Argo Tunnel. It's super easy and the big benefit is, that you do not need to open port 80 and 443 for inbound traffic. I can also recommend to use an AWS Lightsail instances - they are cheap, fast ...
@jackfletch20012 жыл бұрын
1. Identify 2. Probable Cause 3. Test 4. Implement 5. Verify 6. Document A+ drilled these steps into my head.
@Ch1spy42 жыл бұрын
"I'm not an idiot" Red Shirt Jeff edited in "Debatable" I bet lmao
@reggiep752 жыл бұрын
I have to say I laughed at '..maybe I shouldn't have tempted fate..' part. You just know people would've thought 'Yeah, we gotta see if we can beat this website into submission!'.
@TarisRedwing2 жыл бұрын
Wow that motoring and alert system you have was key. You'd have been dead in the water without that goodness.
@Tim-Kaa2 жыл бұрын
Thanks, very useful
@JonDoe-gi5zf2 жыл бұрын
This video is very informative.
@ianallaway49642 жыл бұрын
Hi Jeff, I'm not sure if you're aware but the Cloudflare tunnel client (cloudflared) is actually an open source project. It would also help fix your CG-NAT problem. I've not used kubernetes but dabbled with docker swarm for a little while so I'm assuming it behave in a similar fashion. On the swarm you can run an instance of cloudflared on each node meaning (as long as every pi has an internet connection) you're no longer reliant on 1 node for the ssh tunnel. Love the channel. Thanks
@Star-xf8rd2 жыл бұрын
For Network monitoring I can recommend PRRG it’s quite powerful with lots of sensors prebuilt, you can even create own sensors for numerous systems
@FlygisTheFlygis2 жыл бұрын
“How I survived a sneeze attack” up next on this channel. So glad you’re alive bro
@phlizneinbleedblop23182 жыл бұрын
Woo Jeff! thanks for the info hopefully we all can prepare for inevitable attack
@driodeiros2 жыл бұрын
Thank you for your videos and content Jeff, I find them helpful and entertaining. This video inspired me to improve my monitoring stack (prometheus + grafana also) but it also made me think about the backup/restoring strategies, particularly for your prometheus time series data and the grafana alerts and dashboards. What strategy do you follow to backup and restore those in your environment? Thank you so much again and keep up the good work!
@arekx2 жыл бұрын
Previous video was talking only about frontend nginx on VPS - doing caching etc. So what was php-fpm doing on that VPS and eating resources in ddos case? It was supposed to be nginx only doing cache and pushing things to backend on raspberry pi cluster. So no php needed at all on vps.
@JeffGeerling2 жыл бұрын
Indeed it was, but quickly after the DDoS started, I moved the database back to the VPS and tried transitioning Drupal's traffic to it. To make that happen quickly, I had set up Nginx on that server to still direct _some_ types of requests to the main VPS instead of back at the Pi cluster. Honestly, I should probably take a deeper look at the logs though, because I am also surprised so many requests were hitting the VPS's PHP handler while the rest were hitting the backend. I wonder if it could've been related to an http vs https configuration error on my end.
@cheetobambito97242 жыл бұрын
you know what all these ddos attacks after years of never having a single one? YOURE MOVING UP IN THE TECH LIFE AND SHOULD BE PROUD!(: Good stuff Jeff cant wait for your next video
@luvxinh2 жыл бұрын
Interesting. I was also thinking of what you've been sharing with the world in the spirit of open source. It's basically the documentation for your infrastructure made available to the public
@johncnorris2 жыл бұрын
Sounds like a tough day at the office but at least you've learned a lot about defensive measures.
@nhalliday892 жыл бұрын
Wow isn't that a trip... .It appears that you have been super busy with this madness honestly i envy your patients with how the situation has presented itself...
@DAVIDGREGORYKERR Жыл бұрын
I hope you have your PI Cluster behind a Pf-sense Router to stop virus attacks and other types of attack.
@maartentoors2 жыл бұрын
A "smart' device is as smart as its operator/firmware-updates, even then it can host backdoors (which can come to light using network monitoring tools). I really like your transparency/OpenSource mentality
@agikarasugi22946 ай бұрын
It’s sad that most of the DDoS traffics are from my country. A lot of people here use pirated and questionable software on both their phone and laptops without checking it first. I had my home ISP address blocked, and later found out that one of my family member’s laptop were infected with trojan and participating in DDoS.