The Worst Hack Ever Almost Just Happened

Sponsored: Discover the new @Bitdefender Cryptomining Protection. It’s available for Bitdefender Total Security, Premium Security, and Ultimate Security protection plans at no additional cost for new and existing customers. For more information visit: www.bitdefender.com/solutions...
▼ Time Stamps: ▼
0:00 - Intro
0:26 - The Discovery
1:08 - The Targeted Software
1:58 - A Very Good Thing
3:13 - How It Started
5:57 - The Attack Finally Begins
7:58 - The Hackers Are Forced To Hurry
8:45 - The Full Implications
10:29 - What Else Is Out There?
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
• My Gear & Equipment ⇨ kit.co/ThioJoe
• Merch ⇨ teespring.com/stores/thiojoe
• My Desktop Wallpapers ⇨ thiojoe.art/
⇨ / thiojoe
⇨ / thiojoe
⇨ / thiojoetv
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

[@CentreMetre]

Imagine how pissed off that guy who put the backdoor in is, years of work gone, all cos some guy wanted a fraction of a percentage more performance

[@ThioJoe]

He was definitely punching the air

[@Locomaid]

It won’t be the only one he’s working on…

[@volvo09]

It's scary to wonder how many hidden backdoors are out there, but remain unused.

[@patfre]

Actually it wasn’t a faction of a percent it was drastically slower than it should be. If I recall it was like .5s slower than it should be which is a lot in the computer world

[@goiterlanternbase]

Imagine having foreseen this and loosing one of 300 similar backdoors😉

[@blikthepro972]

remember: the best backdoor is already running, is everywhere, and no one knows about it

[@TomNook.]

Except the NSA / MSS / FSB / Unit 8200

[@scrapmine]

Its called braking in irl. (This is a joke youtube, pls no ban)

[@dokchampa9324]

Ah, fearmongering, my favorite

[@LostShadowGD]

The virus Microsoft puts in win 11

[@GHaKKt]

Humans..

[@gosnooky]

Moral of the story is never come between a database engineer and performance.

[@Stratelier]

It is amusing that, for benchmarking purposes, the engineer who found it was sending SSH requests that shouldn't even pass a sanity check ("wrong username, etc") which explains why he got suspicious of some excess cpu cycles so quickly.

[@Lollllllz]

If he could be working on windows' explorer/taskmanager instead 11's wouldn't be slow as it is.

[@ivok9846]

@@Lollllllz usually one can keep taskman on all the time. not on win11

[@stroodlepup]

@@Stratelier lmao

[@Ascendor81]

I must now change my password from "1234" to "12345" to protect myself.

[@samuelhulme8347]

Technically no matter how strong your password is this back door completely bypasses all passwords because it injects the hacker’s ssh keys onto the infected device.

[@Jonesy1701]

@@samuelhulme8347 I remember my first joke too...

[@NotSoMuchFrankly]

How did you know what my password was?🤔🧐

[@SereneStrategist-kk7mk]

I don't know how you got my passwords but you don't scare me I already changed it into something more secure. With six digits it almost impossible to guess mynew one.

[@samuelhulme8347]

@@SereneStrategist-kk7mk is it “123456”?

[@DavidM2002]

To quote that old adage, "You have to be good all of the time. They only have to be lucky once."

[@dekeonus]

I'm going to have to say: it's not an adage, it was a (very real) threat to Margaret Thatcher. It's still applicable in this case, just a better phrasing might have been: I'm reminded of the IRA's threat to Thatcher: "Today we were unlucky, but remember we only have to be lucky once - you will have to be lucky always."

[@NinjaRunningWild]

You don't have to be good all the time*

[@Jonesy1701]

@@NinjaRunningWild No I think he was correct. We (the good guys) gotta be good all the time, they (the attackers) only gotta be lucky once.

[@JiggyJones0]

​@@NinjaRunningWildpoint: You

[@D.von.N]

I think Seytonic covered this a month ago. But it doesn't hurt to remind ourselves: 1. Social engineering is a thing, 2. Pay developers what they are worth.

[@NigelTolley]

I don't think anyone ever gave the guy any money at all. Then he gave up, and the bad actor(s) took over.

[@Fircasice]

How are you going to pay software engineers working on open source software for free?

[@D.von.N]

@@Fircasice Many so called free software are open to donations. And people donate. Some of the money could be paid to the developers.

[@ottergauze]

The fact this was just discovered by chance really brings into question how many other packages have similar backdoors. This is the kind of stuff that should spur a major investigation.

[@321Jarn]

​@An_EqualNot the FBI or CIA obviously, one of the founders of telegram said the FBI was trying to trick him into using open source libraries for telegram.

[@ottergauze]

@An_Equal Beats me, but it's probably not just gonna be one singular entity.

[@I.____.....__...__]

Like Andreas said himself, this was just incredibly lucky, just a massive coincidence that he happened by chance to be in the perfect position to find it (and _just barely_ in time). A confluence of events like this rarely happens, so it's possible that there is indeed a lot of stuff going undetected. 😕

[@mega_gamer93]

There is a "major investigation". This backdoor has sparked discussion on how to prevent something similar from happening again, made some free software contributors try to audit other software and once again demonstrated the absurdity of a "software supply chain" where the companies don't pay a dime to their "suppliers" yet expect them to do the most rigorous work to avoid hurting their (the corporations) bottom line

[@uponeric36]

@@internet6ate9 Ok, post proof then.

[@jasonlittle6542]

This is the biggest weakness with OSS, but also the greatest strength of it. Anyone can worm their way into a seemingly innocuous part of the Linux ecosystem and taint it. But also anyone and everyone can topple years of nefarious actions through simple curiosity.

[@squirlmy]

the fact this got caught, while Windows CVEs get put out and many admins don't update, leaving vulnerabilities in place for years! Notpetya took advantage of an years old vulnerability in Windows, and caused over $11 billion globally

[@TheGreatAtario]

By the same token, it's not hard for a nation-state entity to get an agent hired at a private software company

[@mr.purger9185]

Bro is flexing proper subtitles 😎 my guy

[@balsalmalberto8086]

Time stamps, subtitles. Bro is a role model

[@staffeyxneyx8420]

Invaluable

[@MrDowntemp0]

The whole linux sphere has been talking about this a lot, but yeah, I think you're the first tech channel with a more general focus I've seen bring it up.

[@UmVtCg]

Not just the linux bubble, the whole Cyber Community.

[@MrDowntemp0]

@@UmVtCg I bet you're probably right, I just don't tend to haunt that corner of the net.

[@marcellkovacs5452]

@@UmVtCg I wouldn't say I'm in the Linux sphere and I'm definitely not in the cyber community and I still knew about it. It was pretty much impossible not to hear about it if you're in the "IT scene" in any capacity.

[@squirlmy]

@@marcellkovacs5452 Its irritating the title is "Almost Just Happened"! no, over a month an a half ago. Clickbait.

[@mdmackint]

Two Thio videos in one day is a win in my books

[@8yt3]

правда радует

[@kuzeyrl]

yess

[@MetsLand]

Agreed

[@balsalmalberto8086]

That's what SHE said!

[@LeoMakesStupidThings]

real

[@WindowsAurora]

The xz backdoor story is crazy.

[@andrewwatson5324]

Not so crazy when you consider that at some point some one probably got themselves hired in order to put in the Juniper back door. This was found about 10 years ago.

[@theRPGmaster]

As a software developer, I have no doubts that this kind of vulnerability (probably multiple) is already deployed everywhere, undetected. Never underestimate the power of social engineering, and these attacks being very easy to miss. Also I remember when ThioJoe had very few subscribers, I'm delighted to see the channel grow like this. I wonder if he remembers me 🤔

[@NotSoMuchFrankly]

Probably like Pegasus on every phone.

[@ronin36963]

Drive-By Mining. You have to give these guys credit for being innovative.

[@KaldekBoch]

As someone on the defensive line working at scale (170,000 users), you do what you can with the control that you've *got* to avoid these issues, but you are mostly at the mercy of others. Where you *really* need to focus your efforts as a defender is being able to detect *when* you've been breached. Our goals are pretty clear - detect within 10 minutes, contain within 60 minutes. That's how fast you need to be, and some would argue that's not fast enough.

[@locinolacolino1302]

My Dad's mate was managing server infrastructure at a hosting company around 2010, and decided to deploy a crypto miner as a cheeky experiment for his team. It was a bit after a fortnight when the team found out, and they chewed him out for misusing company resources, but he immediately returned the blame to them. 'You're saying, if there was actually a piece of malicious software running on our systems, it'd take you two weeks before anyone realizes something's wrong?'

[@_EmptyBox_]

Nothing had made it to the news where I live regarding this. Some tech channels on YT I follow covered the bare bones when this was first discovered, yet the background you've provided has created such a broader and more chilling account of what was really happening.

[@ChrisHeatonbigears5000]

I love your coverage on topics like this. I find it so interesting and you do a great job of explaining the process. Great video.

[@sunla]

We've gotten so spoiled with our technology, we need more code and more programs and more features to cover every base. Thing is, the more we have, the more hands and minds work on the code that run on our machines. That definitely comes with its risks. The truly scary thing to think about is that... logic dictates that the worst is yet to come.

[@EmilyS-gk3st]

And thing is, we can live without most of it, too. Our ancestors even 200 years ago did.

[@Monius13]

Hey, I just want to take a second to say thank you and congrats. I found you ages ago through all the pranks. Was funny at the time, but I can see why you moved away from it. Over time, you've given us some really amazing videos that are very informative and make it easy to digest for those who are less educated on tech. Thank you for the years of entertainment and information and congrats on how far you've come. Much love, bro.

[@brianc5788]

Excellent info & video. keep up the good work!!!!

[@yesterdaysrose5446]

Remember: As an open source maintainer, you should keep an eye on the stuff coming in and just not accept incoming stuff if you don't know WTF it even DOES. (That's the technical term.) But I also realise that if you have relinquished the nominal control to someone else, you're not culpable.

[@Derpingtonshere]

I fully agree with this, but the problem was these so called contributors were intentionally bringing up so called "problems" causing the developer to burn out. Nobody remembers that these people do all this without guaranteed pay, they volunteer their time to better the open source atmosphere. It's really sad that people have to take advantage of good hearted people like this. This is why I always chip a few dollars their way whenever I can. We should try and keep these people happiness high. The actual owner was on Hiatus and gave the reigns to a person he thought he could trust, well that person was taking advantage of his burnout.

[@dputra]

My first contribution was the Harvard's cs50 class CLI tool, translating it to my language Indonesian so my high school students can use it more easily. The maintainer raised this exact issue, "how do we know he pushed something legit, not troll translations?" That's how I realized that while open source contribution is a cool way to collaborate, some people might have malicious intentions and maintainers should try their best to prevent it.

[@NotAghostSpeedruns]

@@dputra They could probably chuck it into deepl translate and most of it would make sense. Having a native speaker translating seems like an improvement over any automated translations though.

[@dputra]

@@NotAghostSpeedruns deepl is not even there yet at the time, only google translate which sucks at translating indonesian to english.

[@Claren.c]

Scary... I needed to check this

[@MichaelGrundler]

At first I thought this video is quite a bit late. I've already seen multiple videos about this backdoor right around the time it was discovered. However I'm glad I watched till the end because this video provided some additional information and context I didn't know of yet.

[@anstropleuton]

I did not expect it to be a topic of XZ... thought this video was some windows thing Also yeah way late

[@benway23]

Thank you for your work.

[@diamondblack3776]

You can do your own micro benchmarking and analysis with Process Monitor from sysinternals and run it as administrator. Picks up background accessing.

[@frankintx699]

Thanks, ThioJoe

[@ORANOID]

Love the format and the story itself.

[@delta_cosmic]

2:20 norton disliked this video

[@volvo09]

Haha, I couldn't believe they tried that. What a scummy company.

[@milentoshev8409]

@@volvo09 What are you referring to? What did they try?

[@Sarah-3]

@@milentoshev8409 Their antivirus software became the virus. Granted it was opt in but there were multiple popups urging you to opt in telling you how great crypto is. They failed to mention about the wear and tear of hardware and the performance impact on other tasks. To top it all of they would not only skip paying the electricity bill they also took a 15% cut from your earnings

[@andreobarros]

@@milentoshev8409 I dont remember all the details, but norton or one of their products had or has a crypto miner within them. They stealthily made it opt-in by default, and when found out tried some justification.

[@AndrewYac]

@@milentoshev8409 Norton tried to install crypto miners in their software without making it clear in the install process lol

[@rodrirm]

First time I heard about this, thank you for sharing.

[@Graham6410]

Wouldn't be surprised if this has happened to other bits of open source software at some point.

[@I.____.....__...__]

Like the xkcd comic Joe showed said, there are a LOT of bits of archaic code that underlie the world's software. We've seen cases where half the Internet broke because software relies on a single function that someone wrote for themselves 25 years and everybody copied. Software is more fragile than people would like to think.

[@BombadilBeardie]

Explained very well. Heard from another youtuber but he made it all the way more complex

[@HKlink]

I'd heard of this, but not the full story. Essentially just heard "some guy was drag racing his computer for fun and noticed a tiny inefficiency which was a brand new back door, catastrophe prevented" and not all the cool details you gave! Thanks for this video.

[@5argetech56]

Zoinks!! Wow Scooby that was a close one.. Whew!

[@cheesepizza98]

>>>>>>>>Is this>>>>>>>>>>>>>

[@jimmeade2976]

I had heard about this, in general terms. Thanks for a detailed explanation.

[@Nadia1989]

The mantainer needs acknowledgement too. Having a life helped to deter the attack.

[@TravisPluss]

You are a saint for documenting what will be the history of tech.

[@wisteela]

When this happened it got me thinking maybe it's time for a big code audit?

[@_SJ]

Wow ThioJoe. Second video for today. I ❤ it

[@disegnosys]

Great explanation and I heard about this threat about a month ago on another channel.

[@brownjames112]

Heard about this the other day on the 2.5 Admins Podcast and the Late Night Linux Podcast, good to hear from some other people. It's a pretty big deal.

[@bruce-le-smith]

thanks for breaking that down, very interesting

[@likebot.]

Yes, I have heard of it before. I watch a few IT channels. Your unscheduled video yesterday reminded me about this backdoor in that it looks like certain actors are attempting long cons to create vulnerabilities.

[@doge7831]

This backdoor only affected amd64 systems (so ARM computers wouldn’t have been affected) and it would likely take some time before it got into Debian and Ubuntu LTS (used by a ton of servers), as they only receive non-security updates every ~2 years, so if it was discovered 1 month later, we would probably be fine.

[@user-28qhfk65]

1. If I remember correctly, there's a code that check specifically for amd64 (and x86?) architecture for it to run. (sus imo) 2. We're very lucky that the backdoor was found before it was released into stable Ubuntu LTS 24.04 release on April 2024. That might be the attacker's main target. 3. The fact that it was found by coincidence by microbenchmarking, ~500ms delay, is very concerning. 4. The attacker will learn from this mistake and might pull something like this again / another party is inspired by this move will do it in the future.

[@nicholasvinen]

For some value of "fine". Yes it wouldn't have been a disaster but some servers would have been compromised for some time.

[@ca_kay]

This guy deserves a medal.

[@russian_alex]

idk what to comment (nice video thio, keep up the good job)

[@dualbeardedtech]

I had heard about it but, like you said, it was only from tech news outlets. Thank you for making a bid about this!

[@Norman_Fleming]

Was aware of this but good it is still getting coverage. Really feels like this house of cards is not gonna stay up much longer.

[@REMY.C.]

I'm not a programmer and I immediately spotted the "." because I don't like when it's not tidy 😂

[@DudeSoWin]

Performance is number one Safety Third

[@TomNook.]

Jia Tan is a Chinese name, Jigar Kumar is an Indian name. People who want to stay anonymous won't use their names, but also doesn't want to introduce a rival nation to investigate (so he didn't use a german name for example), so quite likely a hacker of russian origin. Isn't geopolitics wonderful.

[@Reddotzebra]

So the backdoor would likely quietly delete itself if it detected a Russian keyboard. Making it legal for your citizens to attack any system as long as it's not one your nation owns is a stroke of genius, ngl. I wonder how much money they've saved on buying day zero exploits from the usual sources?

[@NinjaRunningWild]

Non-sequitur. Nothing can be deduced from the name.

[@mega_gamer93]

'Cheng" is a cantonese name while "Jia" isn't. This indicates however made the backdoor just tried to think of a name that sounds Chinese enough. Such sloppiness is typical of the US But trying to deduce the perpetrator from the name is stupid anyway, we could go in circles all day talking about potential 5d chess by the perpetrator

[@MiseRaen]

​@@mega_gamer93The OP just have the politics brainrot. The culprits might be multinational anyways.

[@shanent5793]

"Gee-yah" isn't Chinese, it only looks that way. In Chinese it's only one syllable.

[@kyouhyung]

The method and approach they took were very refined and systematical. I wouldn't be surprised if this was only one of the many similar attacks.

[@wangel81]

As an IT Professional for over 30+ years, the world is a SCARY place. I just wanna go live in the mountains. Tired of playing whack-a-mole all the time. No matter what we do, there's always someone or something else out there that is better.

[@AraiDigital]

“And I would have gotten away with it if it weren’t for that benchmarking kid!!”

[@RobTheMusician1]

Those antivirus softwares are useless. We need more performance tweakers.

[@BigWhoopZH]

Warns about one malware, advertises another.

[@benyomovod6904]

I bet the NSA planted the ultimate backdoor into silicon long ago. It is absolute logical

[@nanopi]

I knew about it during the Easter weekend thanks to a general channel Discord and some Linux/programming youtubers. Arch really quickly updated the package and posted to their news page when discovered. Also just happened to be 1 week after I updated Arch WSL for a Samba setup involving Windows 98. OpenSSH does not normally use liblzma but got patched by Debian/Fedora/systemd systems to work with libsystemd which did use liblzma but then 7:58 this pull request was going to make systemd not automatically load liblzma all the time which pretty much doomed the backdoor.

[@akimezra7178]

I remember it when it was just discovered, but i think your video lacks the explanation of just how BRILIANT this backdoor is, and the code behind it.

[@TymexComputing]

7:07 - very nice idea - must try it sometimes :)

[@TymislawMiau]

I love your vids

[@MonteVanNortwick]

Yup. How could we know if backdoors already have been installed? Until...they are discovered. It could be a million or it could be zero.

[@aylivex]

I knew about the backdoor since the time it was discovered in March 2024. The backdoor was discussed, it seemed, everywhere, I also watched a few videos which explained what it was and the consequences if it weren't discovered in time.

[@Its-Just-Zip]

This is an excellent demonstration of both the benefits to security that open source software has as well as the threat to it. Had this been a closed source project this back door had it been implemented would never have been caught but also it would have been much harder to implement.

[@vlc-cosplayer]

"it would have been much harder to implement." -- Yeah, it'd be really hard for a 3-letter agency to give that company generous funding, in exchange for a backdoor... ;3 And in case they refuse, remind them that accidents do happen...

[@Mmouse_]

This was an amazing idea...

[@verzagen7550]

So one thing to add, this was included in rolling release ditros like Arch as well, but my understanding is that, the way Arch used xz and the way Red Hat and Ubuntu used it were different enough that it wouldn't actually effect Arch systems

[@keit99]

Arch doesn't have ssh compiled against liblzma (which debian and co had)

[@verzagen7550]

@@keit99 thanks for adding, couldn't remember exactly why Arch was different from the rest in this regard

[@mega_gamer93]

​@@keit99the distros did not link sshd with liblzma. The distros patched sshd on a way that linked it to libsystemd which is then linked to libzma

[@keit99]

@@mega_gamer93 right that was it. IT's been a while since I read about the backdoor properly.

[@zalyster]

Crazy story, even crazier to think about what's out there that we don't know about.

[@tonyvn5817]

I hope Windows 10 never gets hit by backdoor hackers O.O Totally enjoy your video TJ. Peace brother.

[@dave_dennis]

I did hear about this through mainstream media but I don’t remember where. I do recall the detail of it being discovered at Microsoft but there was no more detail than that.

[@JB52520]

It's kind of reassuring that software improvements make backdoors in other projects obsolete. If the good guys keep fighting, they can plug holes they don't even know about.

[@NinjaRunningWild]

Low Level Learning covered this right after discovery. His video is also worth watching.

[@nou712]

He also said it's the end of open source and linux.

[@user-hg7qw3pl7o]

I already knew this because i have subscribed to morning brew!

[@adriany4700]

Scary stuff

[@cheeseparis1]

I once thought about open source risks and then forgot, since a whole community watches what changes... but if the hacker is patient (2 years!), it finally can happen... No I'm not afraid at all. _casually updates USB hub firmware, nothing bad can happen_

[@Bruno_Haible]

9:37 The pictogram for SSH is insulting.

[@EnergizerTX]

It was discussed by Dave's garage on April 4, and other sources as well.

[@shackcf]

I did read about this just after it was revealed by the Microsoft employee. I think I found it in one of my Flipboard items. However I did not have all the details that you just spoke about.

[@timothyvaher2421]

Theo, you do a cold impression of a North Korean Cyber 🪖 Officer.!

[@xX_ang3Lz]

this is crazy i remember you telling me to tape batteries to my cat5 to make my internet go faster

[@jaybingham3711]

Nothing worse than not finding the backdoor!

[@Sypaka]

Don't forget, this incident caused people to scan all their code for more backdoors and fixed them, making Linux even more safe.

[@eldibs]

This would make me paranoid about security if I wasn't already paranoid from the time I (temporarily, to test something) opened up SSH access over the internet to a Linux machine on my network and saw it immediately get hit with constant brute-force login attempts.

[@tstager1978]

I did hear about a problem with vulnerability with the xz software but it wasn't widely covered.

[@LeeMaiden]

Bitdefender is good stuff, I have Win 10 on two desktops, but I'm usually on either machine using the main hard drive which one is Linux Mint's LMDE 6, and the other machine is Linux Mint's 21.3 Cinnamon. I have Bitdefender on my Mini PC that has Win 11 Pro on it, I have Bitdefender on my phone also, it's one I recommend to clients of mine also. It's a shame they don't make a version for Linux that would have caught this XZ problem. I rarely use Windows, so this one hit close to home hitting Linux with a well used tool.

[@ranxlusactualmainaccount]

ThioJoe got a 2 vid in a day Streak

[@QuotePilgrim]

XZ Utils is not really a behind the scenes thing for most Linux users though. Almost every Linux user will come across and extract .tar.xz files regularly, and sometimes even compress their own xz files, both of which require XZ Utils. In other words most Linux users are aware of xz and use it frequently.

[@BrianG61UK]

This was discussed on the Security Now podcast episode 968 on 2nd April.

[@DeadKoby]

Super cool... an OCD technician wants to make the performance ideal, and thus saves the day.

[@Lampe2020]

Yay, another video about the xz backdoor…

[@TrimutiusToo]

I heard about it a couple of times before... But i fall several tech specific channels that talk about stuff like this all the time

[@Blues.Fusion]

Someone needs to write a screenplay for this and make a movie. So much drama.

[@kubakielbasa5987]

If it doesn't happen until I'm a multibillionaire then I'll make sure to organise it and make the movie free-to-watch and free-to-pirate.

[@soulman902]

I saw this with another OSS project where Bitcoin mining was added to a library which was being used by a commercial project which was used by the company I used to work at. Our Anti-virus caught it being installed by the Dev and the company that put out the update had to release a new update.

[@realMrVent]

Ugh, this is just gonna be a feast for proponents of closed-source...

[@mrfoodarama]

This is an awesome story! Important as well, not much was mentioned about it to normies. The only things I saw in the avg person scope was trying to use this as a way to badmouth open-source software and that really did not sit well with me

[@mega_gamer93]

Bit defender endpoint for Linux is for file hosting servers to check if the files they are hosting are malicious, it's not for self protection

[@peter-uy2iv]

as a owner of few public servers, i almost shitted myself

[@edalder2000]

Wow. Used as intended, this is scary.

[@evertonshorts9376]

Reflections on trusting trust.

[@MalwareLab150]

I love your video ❤❤

[@princess_wawa]

i remembered how much of a nerd i am when he said there wasn't much coverage about it but all my news feed was about it

[@FlyRetroGamer]

Please note that Ubuntu is Debain under the hood.

[@itchioisshrt-jd3vl]

hey thiojoe i have a question, if i delete the microsoft folder in registry editor , will it brick my pc?