Linux got wrecked by backdoor attack

A popular compression library called XZ Utils was recently backdoored by a hacker which compromised Linux distros like Debian, OpenSUSE, Fedora, and Kali. Learn how the liblzma hack happened who is behind it.
#programming #linux #thecodereport
💬 Chat with Me on Discord
/ discord
🔗 Resources
Details of XZ vulnerability www.openwall.com/lists/oss-se...
CVE-2024-3094 access.redhat.com/security/cv...
Weird Algorithms • 10 weird algorithms
Cryptography Tutorial • 7 Cryptography Concept...
🔥 Get More Content - Upgrade to PRO
Upgrade at fireship.io/pro
Use code YT25 for 25% off PRO access
🎨 My Editor Settings
- Atom One Dark
- vscode-icons
- Fira Code Font
🔖 Topics Covered
Overview of cve-2024-3094
Can Linux be hacked?
Who is behind XZ backdoor attack?
Home does XZ backdoor work?
Worst hacking incidents of 2024
Which Linux distros were affected by XZ attack?

[@uplink-on-yt]

Thank you for reporting this bug. The next version of the backdoor will no longer slow down your SSH server.

[@squarerootof2]

Next version will make SSH faster.

[@PrathamInCloud]

@@squarerootof2 It would be so funny if true

[@immortalroasterben7146]

Fbi knows your location my italian friend

[@alphabee8171]

or open source communities will be more "careful", security is always catching up with the bad guy since you're defining rules of the game and they have to bend those rules and then you make a new rule with a patch and this goes on and on.

[@futuza]

​@@squarerootof2for once the NSA will be putting it's resource to use in trying to help the people

[@cheezyskipper]

Imagine planning this attack for 2 years just for someone to find it by accident because their CPU was 500 ms slower

[@pbsuite]

Bruh.... I would have gotten away with it , if it wasn't for you meddling software engineer, bro is built different ...

[@dtmt502]

this is not just the one there are many others planted everywhere cooking

[@gg-gn3re]

yea and it was found in 3 days too.

[@6IGNITION9]

Humans can detect 13ms of latency. This was ~40x more than that.

[@xeqqail3546]

Imagine you normally send some 1000 files to server and every file took 100 ms, it will only take 100 sec. But you noticed that it gone up to 500 sec. That's pretty sus 500 ms increase in benchmark

[@Dira_1111]

Attacker :- Plans for years to attack 🤡 Our guy :- CPU took too long (500 ms) , I must check 🗿

[@ItsRyanStudios]

Lmfao is that a Chad emoji? 🤣

[@prakash_77]

@@ItsRyanStudios It's called 'moyai'. Very popular on Discord.

[@apIthletIcc]

Indeed it is

[@_________________404]

The based chudcel be like: "SSH has fallen, millions must investigate"

[@damianm-nordhorn116]

​@@ItsRyanStudios Moai.. Haven't you played Civ V?! ;)

[@NeunEinser]

The guy helping to renovate the apartments, hiding cameras which are only caught because a slight increase in the electricity bill is such an amazingly good analogy, well done!

[@meepk633]

Is it? No one looks for small increases in power draw. Tens of thousands of people and bots look at benchmarks for common operations like SSH logins.

[@RealNaisuCinema]

@@meepk633that’s literally what makes it a good analogy if you had been listening lmao. ‘No one’ looks for a slight difference in CPU usage on startup either. Thats why so many people were vulnerable to it. This one guy just so happened to look into it. Just like out of all the neighbors that one guy just so happened to look into it. It was a negligible difference and he still looked into it.

[@meepk633]

​@@RealNaisuCinema This backdoor was caught as soon as people started incorporating the compromised liblzma updates. Hardly anyone was vulnerable to it because of how quickly it was discovered. It was discovered quickly because people profile and test their apps continuously. Dumb luck was not required. He noticed the extra 600ms of latency and other fails on *every* SSH login. He looked for changes, found them, and determined what caused them. The camera analogy is stupid.

[@Robert-cc3wr]

@@meepk633 you're a sad, sad hater. I feel sorry for you

[@amarissimus29]

@@meepk633 You're assuming a specific audience. Normally, I'm with you; analogies suck. They're used to patch a flawed understanding of the presenter's own knowledge. But the actual exploit was covered decently enough for a short video. The analogy expands it enough to reach an audience lacking your godlike knowledge and skills while keeping a reasonable hold on the core issue. As much as we all aspire to be you, we're just too stupid. Damnit, listen to me, defending analogies, of all things. Thanks for that.

[@jayshartzer844]

A moment of silence for the NSA having lost one of their favorite tools 😔

[@thecodemachine]

Its probably China, Intel Architecture already has a backdoor.

[@MaxPanic]

Their previous attempt at adding a back to Linux was also denied.

[@GSBarlev]

Yeah, some state agency is _extremely pissed_ right now that their op was busted after two years of work, and before their backdoor could actually make it into the wild.

[@DigitalForerunners]

Why do you think the “exploit” has been publicly dropped by Alphabet?

[@SALSN]

​@@MaxPanicone of them anyway 😬

[@Arckil]

Few percent of CPU usage increase and 500ms of additional delay when SSH into a machine ? Sus indeed amiright

[@Paulo27]

Meanwhile when I ask our guys why a server that took 20ms is now taking 20s: "there's no one here... must have been the wind"

[@GSBarlev]

Database programmers are a different breed, my dude.

[@seekingthelovethatgodmeans7648]

Someone built a system utility that was ssh-ing all over the place like mad (the kind of use/abuse of common system utilities that mad system programmers, of which I was one in industry where they allowed it, are wont to design) -- and discovered one day in obsessive testing that its performance on a brand new, not yet stable release of Linux had become a dog. Well, THAT can't be tolerated... and voila, the backdoor setup was outed. The backdoor had given itself away by... irony of ironies... a SIDE CHANNEL, in this case its performance impact. Hurrah for obsessive utility polishers. By a Microsoft developer no less. Now I wish an obsessive Microsoft programmer would fix a rendering problem in Photo that's been around for more than a year and gathered numerous complaints, but again my problem isn't a security problem that could let a malicious actor into systems worldwide.

[@Mordecrox]

That's actually extremely significant if you're doing benchmarks and especially in databases, if an operation takes on average 10ms and now consistently takes 12-13 and this operation runs "all the time as often as possible", you can guess even my non-technical self will put the tinfoil hat and go on a hunt. Fren was going for the one edge case this had the barest chance of being detected

[@seekingthelovethatgodmeans7648]

This adds up when a script is hammering a system with dozens of these over and over.

[@earthling_parth]

I manually came back to check the channel as I didn't notice any AI or tech industry updates via 'The Code Report' in my feed for the past 2 weeks

[@user-fr2jc8xb9g]

yeah , in these moments i realise i'm addicted to fireship videos...hopefully everything's okay with Dylan.

[@macknittle1121]

Well guys, it's been three weeks. They got him.

[@pvc988]

Guy who discovered pretty nasty backdoor because of CPU usage spike that lasts for couple of tenths of a second. Meanwhile, there are so many users that are unable to identify the friggin crypto miner on their system that eats 99% of their CPU/GPU all the time and think that it's just their machine "getting old".

[@jcozyyt]

What are the main culprits of poor computer performance? I've been told if the drive your OS is saved to is close to full that can affect performance, but I'm sure there are a few other causes besides a crypto miner

[@flybyray]

You are so Correct! by watching KZbin videos we all the time run a payload for the bad guys.

[@pvc988]

@@jcozyyt If it's not malware then the machine may be just overheating and throttling to protect itself from permanent damage. A little cleaning, new thermal paste, etc. may help. If slowdowns are really serious (like "random" freezes for a minute or more), then it's often the HDD that's on its way out. SSDs usually fail more abruptly without many early symptoms. If you are getting complete crashes then check the RAM and the power supply.

[@sirseven3]

​@@jcozyytbloatware installed if using windows. There are redundant processes or features that go unused typically (print spooler services, Bluetooth, Cortana, accessibility). Outdated drivers are also a big component of performance issues and vulnerabilities.

[@vladimirmijatovic4171]

That cryptominer in the background is just called Windows xD

[@r1konTheAutomator]

I'm a life long nerd starting in security in middle school and an engineer now at 39. I live and breathe computers. My wife isnt in tech at all - but she just not 5 minutes ago told me about this hack and used the correct terms. I've never been more turned on.

[@Whynot83848]

😂❤ Enjoy the backdoor

[@philippefutureboy7348]

You may have switched the "arouse" binary with the "think" binary my friend. I'd do a check if I were you lol

[@ThomasAndersonPhD]

Did she invite you to attack her backdoor?

[@neuvx]

​@@ThomasAndersonPhD dawgg

[@ITSJTG58]

@@Whynot83848LMAO

[@boltez6507]

Linux backdoor discovered. Every tech youtuber: that's free content.

[@sleepyearth]

It's all good because they help to spread the hack around. This makes people more alert 😊

[@mcipovic]

And he said nothing in this video. Like we are all retarded.

[@vaisakhkm783]

@@sleepyearth :) you meant, give hackers idea and people forget it in 1 month?

[@boltez6507]

@@vaisakhkm783 dude after a vulnerability is discovered its basically of no use for the hacker.

[@vaisakhkm783]

@@boltez6507 no, i meant now people will try get backdoor into other projects too

[@iainballas]

The guy who found this and exposed it needs a medal. He prevented a disaster on the scale of any nuclear meltdown in terms of financial cost and damage to society.

[@shuki1]

Nobel prize or at least some national recognition. This is better than any olympic gold medal.

[@vaisakhkm783]

Temple OS: 0 maintainers, 0 supply chain attacks...

[@RajSingh-gz6mr]

Holy C 🗿

[@69Deez_Nutz69]

Terry laughing at us mortals.

[@Lewdovico]

0 Daily user

[@RajSingh-gz6mr]

@@Lewdovico bcz it's now *only* an relic artefact for people who want to explore OS design from scratch.

[@xTsubasaCrossx]

Reported this loophole to Jia Tan. He is on the way to fix this "missing" backdoor 😂

[@Scratchfan321]

This is why you're supposed to write your own operating system from the ground up

[@alexleo4863]

Yeah, and we will help you maintain it

[@howTo_79]

lol

[@samwalker7567]

Hardware backdoors exist.

[@Scratchfan321]

@@samwalker7567 Just wire the transistors together manually then

[@Pakistani890]

@@Scratchfan321 lol😂😂

[@GigaSimp]

They got him. It's over.

[@glitch933]

Dude u there? 💀

[@RevenantCovenant]

Help me step maintainer, my ssh login has a 500ms delay

[@_tr11]

​@@pyromaniac2359 me: nothing going on here

[@KR4FTW3RK]

@@pyromaniac2359 as a full time M$ admin I can confirm this statement is true.

[@CuteSkyler]

500ms is pretty major for just pure SSH

[@kikc]

0.1ms*@@pyromaniac2359

[@kikc]

Since linux users are so toxic I will NEVER get it and it's YOUR FAULT. @@pyromaniac2359

[@edhahaz]

The real miracle here is how a MICROSOFT employee noticed a slowdown and didn't assume it's from Microsoft improvements.

[@sliker-hq1mt]

XD

[@xeqqail3546]

Its a new update for copilot bro

[@lukarikid9001]

@@xeqqail3546 now with more bloatware and telemetry yippeeeeeeee

[@w1d3r75]

that's a good one 🤣🤣

[@clooood]

excuse me sir, 3 weeks without a code report is getting painful. wish you the best sir

[@vasudevsharma5390]

RIP Fireship

[@kemzops]

The non-technical analogy is insanely accurate 3:29

[@slashtab]

superpower of fireship

[@pbsuite]

Bro... I do that all the time 😢😢

[@ycombinator765]

obv AI

[@MODEST500]

fireship probably - hey chatgpt i want to make a video on this topic explaining this incident to fat devs living in bad neighborhood, give me subsequent scenarios or real life but simplified analogies to help explain

[@sliker-hq1mt]

that why i like fireship

[@HemstitchedIrony]

Its incredible that the security of millions of machines and billions of dollars worth of tech often depends on one random 37 year old polish dude who maintains a core library or utility used by basically everyone.

[@kamu38]

we're screwed.

[@Tubeytime]

"random" because the smartest people in the world don't want the spotlight

[@AtaGunZ]

xkcd 2347

[@sethm7761]

welcome to the internet

[@wesleyhinds7742]

And he doesn't get paid.

[@yesyes-om1po]

where the heck is fireship, i need a new video on all the new AI

[@4RILDIGITAL]

This is really concerning. It's clear how vital it's becoming to ensure the security of open source tools, particularly those that are widely used like XZ. It's scary to think what might have happened if this backdoor hadn't been discovered.

[@kmlau1986]

At the same time, it is all the fact that it is open source that the malicious code can be discovered. The amount of undiscovered backdoors in close source proprietary software can only be magnitudes more.

[@creativecraving]

Or rather, it's becoming clear how vital it always was.

[@creativecraving]

​@@kmlau1986💯 There's not a business justification for searching for backdoors in proprietary software until one is found, or unless you have specific wording in the EULA or if tighter-than-average regulations are involved (e.g HIPAA)

[@angelorosa9534]

Software world is full of backdoors, the only difference is when you notice them

[@elpapito24529]

What coulda happen tho? As in for real what could have the malicious party done with that back door? can someone elaborate

[@The0Yapster]

It is insane how that security expert took the time for such advanced diagnosis on an unstable distro from few subtle symptoms. If that has been me, I would've simply nodded and said something in the lines of : "This is probably because it is an unstable version, they will probably fix it in the stable release" and moved on with my life.

[@celchronicles]

The fun part was that Freud wasn't even an security expert.... He was one of the committer for Postgres Which made his discovery ever more impressive

[@cryingwater]

He probably ran the same benchmarks so many times the discrepancy became obvious

[@creativecraving]

Yeah, since he's a package maintainer, these are probably standard techniques he uses in everyday life. Still, I'm grateful for his dedication.

[@marcinpawelw]

Except the guy who found it was one of the people who fix it for the stable release.

[@daniser87]

what if Freud is undercover counter-intelligence officer of opposing secret agency 😱

[@amitdaniel6327]

Working in cybersecurity a few years now, always overwhelmed to hear how monstrous are some security researchers are, detecting these random vulnerabilities.. impressive

[@advertslaxxor]

This guy said he isn't even a security researcher too :D

[@EwanMarshall]

He wasn't, he was just a software engineer at MS who stumbled across it.

[@kaaaxcreators]

the guy that found it wasnt even a security researcher

[@amitdaniel6327]

Well he’s creative I’ll give em that, probably could make a transition to cybersecurity easily if he liked

[@FireFox64000000]

That's the best part. He was just some random software engineer. And like every engineer he was annoyed by something not being as efficient as he wanted it to be.

[@thealbaniandude1997]

Where is Fireship now?

[@kittyjuicer]

I know nothing about code and to me this video was entirely in alien language. I don't know how I got here or what any of this means but I'm glad nothing bad happened! Thanks Fren!

[@luciusartoriusdante]

haha what a legend.

[@n3bul0n]

funny you still watched

[@dontsueme]

Same here, someone please exlain as if i was a toddler (cuz i am when it comes to computers)

[@n3bul0n]

@@dontsueme The analogy Jeff made at the end with the camera installed in your toilet is a really good explanation

[@akpokemon]

really? even after he dumbed it down with that ridiculous and unnecessary analogy at the end about the landlord?(well...at least I _thought_ it was unnecessary, but people like you apparently are watching)

[@lotfiabdallah5110]

Don't worry, If you are using a stable distro then you are haven't been backdoor-ed nonconsensually

[@luisluna5835]

Not that we are aware of... minor but important distinction.

[@Rundik]

No, that means the backdoors you have haven't been disovered yet

[@christianh2581]

I read the original comment as 'you might have been backdoored consensually' 😉

[@luisluna5835]

@@christianh2581 lol

[@brunopanizzi]

If you use windows/microsoft products you are being backdoored consensually

[@GSBarlev]

To clarify, this attack didn't actually affect any production systems-every stable distro was at least two minor versions old, and no rolling release was built in a way such that to be vulnerable to the attack vector. The *one system* where this was out in the wild was... macOS systems with Homebrew. But don't let Tim Apple find out, or he'll try to block macs from "sideloading" software.

[@LosFarmosCTL]

the exploit doesn’t actually work on macOS though but in general that’s definitely a problem of using rolling release software, the same issue was also technically present in the latest arch release, but from what I’ve read the exploit doesn’t work on arch either

[@GSBarlev]

@@LosFarmosCTL Pretty sure you're right about Homebrew from what I've found-the general consensus appears to be that the FOSS world dodged a bullet by Freund discovering this in March and not May, by which point this version would have actually been deployed in the intended target-Ubuntu 24.04 LTS.

[@Sunshrine2]

This is the better explanation of "If it is not broken, don't fix it" in the computer world.

[@LosFarmosCTL]

@@GSBarlev yeah this feels very much like a long game attack that was supposed to end up in incredibly valuable targets and if they managed to slip it into a stable ubuntu release without anyone noticing… oh boy that could’ve been a disaster would be really interesting to know who was behind this, but since it’s probably some government agency ig we might never know

[@MatheusKlSch]

afaik not only homebrew on mac but also msys2 and cygwin on windows shipped the bad library but quickly reversed to a more trustable version

[@denvermalcolm2838]

dude got taken over by an AI

[@unknown-fd1yz]

Bro, are you okay? You not on earth or something? It's been 12 days and no words from you 😅

[@GursimarSinghMiglani]

Get a life

[@unknown-fd1yz]

@@GursimarSinghMiglani sure, thanks for telling me. Have a good day :)

[@SkittlesWrap]

The problem here is that a side effect caught the malicious code. Not an SSH developer. Not a developer with liblzma dependency. Not the xz developer. A frigging user of ssh just happening to be testing detailed CPU performance. This is as brittle as security can get.

[@seeibe]

To be clear, no openssh implementation uses xz as a dependency. It's these particular distros that patch xz into their ssh implementation. So at the end of the day is that these distros were 100% trusting xz to the point that they patched it into one of the most critical parts of their system, while in the meantime xz was being maintained by a single person who wasn't feeling well enough to really fulfill the role.

[@futuza]

​@@seeibeand yet no one else volunteered to help, other than a state actor with malicious intent, so will anything change in 5 years when, inevitably, some other critical dependency with a single maintainer is also backdoored? No. Security is screwed by our apathy as a species. There are hundreds of other repos out there run by basically one guy, who's asking for help and no one comes because 1) They don't think they're qualified enough. 2) They don't have time/are too lazy to help. 3) They don't help unless they're being compensated somehow but the maintainer had no money to give them. 4) The original maintainer is a brilliant, but autistic asshole who does not play well with others, and thus can't convince others to stick around. 5) They're being overworked by a corporation to work on something else, and have nothing left to give. 6) They're happily ignorant of the precarious wobbly jenga tower our entire technological infrastructure is built on and trust software out of pure natievity. 7) They're aware of the issues and how to fix them but disagree with the maintainer on some design or philosophical difference and are too prideful to reconcile with them, so instead they fork the project and no one has started using the fork yet, because the old one still exists and gets updates. In short, nothing will change, and we keep on living praying that one day our technical debt as a species won't catch up to us.

[@Spartan322]

@@seeibe Which makes it kinda funny that the only distro this specific backdoor wouldn't work on Arch because Arch doesn't do dumb stuff like that instead of requesting upstream enable something that could supplant the patch.

[@whannabi]

​@@Spartan322 I use arch btw

[@theunknownkadath]

It is odd that a closed blob was allowed to ship with the code. But on a diff note, Linux is amazing in that it is designed so one can look deeper into any process.

[@Alcaline-hu2vu]

I think the real take-away here is that we need to make sure that extremely important core libraries aren't maintained by a single dude, because that was the main reason the other dude managed to create a backdoor in the first place

[@ChamplooMusashi]

exactly, these are the kind of targets these groups are seeking out. and the next exploits will only become more complex since they will look at what happened here and understand better how attacks can be obfuscated

[@seeibe]

Companies need to pay for the open source software they use already

[@seekingthelovethatgodmeans7648]

@@ChamplooMusashi Hopefully white hats will keep a step or two ahead. Now that we're more alert about how something like this can happen, deltas between releases will be more thoroughly scrutinized. If something hefty changes or is added with no good explanation that can't be independently verified, the change will be put on hold.

[@jfernandez76]

Tell that to NPM projects 😂

[@jeliasson123]

@@seeibe Why, and which open source license are you referring to?

[@BasDado]

I think we got really lucky that this got discovered, but that makes me more suspicious: if this got as far as a debian-unstable releases, there MUST be similar backdoors in stable releases that just haven't been discovered...

[@matthewsimmons2246]

Thank you very much for the apartment analogy. I'm very uneducated in these subjects, for now, and that really helped me grasp the previous stuff you were explaining :)

[@Napert]

1. If you're not using distro with rolling (unstable) releases, you're safe 2. The backdoor was in xz/liblzma, which official sshd repo does not use, so sshd itself isn't backdoored 3. Some distros patch sshd themselves to support systemd messages, and in that process also link xz/liblzma 4. Affected xz versions are 5.6.0 and 5.6.1, if you have older versions or updated a day ago, then you're safe (xz -V to check, patched/fixed version is 5.6.1-4 iirc)

[@xwinglover]

5.6.1-3 is also safe

[@bowlseriw]

Don't use xz -v directly, find out the version through grep

[@JonnyArmano]

Yup, great summary. @Fireship: You erred on (2), OpenSSH does NOT use liblzma for compression. Kindly clarify that!

[@philippefutureboy7348]

It's really insane how many of core, crucial parts of our technologies rely on single maintainers who do this for fun!

[@Paulo27]

Just takes one bad library...

[@ionrael]

and also they don't get paid while companies makes millions with the work of others

[@seeibe]

@@ionrael This. Once again capitalism is at the root of these problems. Honestly there's not much difference between this backdoor and the bridge that recently collapsed, except that in this case we got lucky.

[@dchri18]

@@seeibe Another example of a surface-level understanding on complex economics. You can't just blame everything on "capitalism". "Human greed" would be a more fitting blight.

[@seeibe]

@dchri18 It's capitalism. Human greed is not the issue. It's the system which rewards particularly greedy individuals and propels them to the top which is the issue.

[@ThemePro24]

The ability for the engineer to discover this backdoor before it was widely shipped was only possible due to the open nature of the project.

[@raphaelcardoso7927]

I'm happy that open source also helps figuring out what the hell happened to find and fix backdoors. Thanks as always Fireship xD switching to templeOS right now by the way

[@EwanMarshall]

-One small mistake, sshd does not depend on or use liblzma, instead some distros are patching it to link to systemd for systemd-notify and systemd uses liblzma, openssh are in their own implementation of adding a way to use the interface without actually linking in systemd to not increase the attack area and systemd are currently working on pairing down their dependencies and isolating what is needed between components, both these have had patches to those ends in the last few weeks before the discovery and publication of this backdoor. Also means the attack doesn't actually work on arch linux, the question we really have is did Jian Tan sneak something else in somewhere in something they touched.

[@MatheusKlSch]

afaik Jia Tan also contributed to libarchive. They also maintained a unit testing library for C. People are scrambling code and trying to remove any of his contributions.

[@ArneBab]

Learning that systemd is what exposed the distros to the problem, because it sidestepped the dependency checking done by OpenSSH folks … wow.

[@marsimplodation]

why does the attack not work on arch? It uses systemd as well by default, with a opened ssh server the system should be vulnerable, right?

[@EwanMarshall]

@@marsimplodationbecause arch do not apply the patch to openssh to link it to systemd-notify, if just doesn't pass tell systemd about status changes of the running daemon.

[@EwanMarshall]

@@MatheusKlSchYes, several projects they contributed to, part of how sophisticated this is. Why I think it is state sponsored, don't know which state though.

[@ohheyos]

The title 😭

[@BoogieBeatz-5]

😂😂😂

oil up, be there at 8

[@TuxikCE]

what is up with the title? Did it change?

[@jc918a-32]

Ass wrecked

[@hamadaelwarky3640]

"non consensual backdoor attack" 💀💀💀💀💀​@@TuxikCE

[@danieltober8574]

how do you manage to make these videos so dense with information while still making them hilarious? so good

[@serhiirudenko6183]

Thanks for explaining this. I was waiting such video because I had no idea what those memes about on tweetor.

[@shivamshivanshu5682]

If this is a long planned organized attack on Linux, you can only imagine how many such backdoors could be present in Windows/ Mac at this moment, someone joining as a trustworthy employee, working over years pushing such malicious code with no chance of detection since the source code is not public. Scary

[@Binxalot]

Its probably this many kzbin.info/www/bejne/mZ2TYYN-prNonKc

[@octia2817]

Even this issue was barely detected. We got REALLY lucky. I wonder where else there is malicious code like this?

[@l3p3]

These companies have strict review rules for this. There is no code published from MS that is not reviewed, they even have a dedicated security review I think.

[@jaideepshekhar4621]

I thought my company also had "proper policy" and "strict reviews". 😉

[@Leonhart_93]

If the software is open source with very few contributors, it's more likely.

[@MaZe741]

NSA be like, "Yeah but WHO would notice a half-second CPU spike during a hidden build process" German Fren: Isn't it odd how...

[@GSBarlev]

Counterpoint: I'm surprised the NSA didn't notice this themselves: "Hey, there's this weird 500ms slowdown in our botnet playbooks. Someone needs to dig into that."

[@edilgin622]

@@GSBarlev maybe because they are behind it?

[@martinverbeek5214]

The AI got him 😢

[@H4KnSL4K]

Nicely done commentary (with images) on this issue!

[@noahvandal6485]

God mode programmer skills to be able to detect that

[@michaelsills8038]

More like god tier observation level haki.

[@bladman9700]

@@michaelsills8038 more like autistic tier observation. blud got mad at the .020230248293 mili seconds of delay

[@GSBarlev]

This 🐐 develops postgres for a living. With all the db exploits he's probably seen over the years, this was probably a giant snooze-fest for him.

[@anatolydyatlov963]

Making it was equally impressive, though. That level of dedication is really inspiring

[@gokhanersumer2273]

Not really, several years ago my desktop computer got infected once and I noticed something was wrong right away because of slowness. Btw, I'm just a mediocre level programmer. This is the same. One developer noticed SSH was being slow and investigated it.

[@mitchellmnr]

The way the attach actually happens during the build process is extremely well done ... kinda wish you went a bit more over that instead of a skim .... but most people dont really care so I get it haha

[@clawwer4404]

This channel doesn't really go in detail. But reports on important/cool stuff in a short format way. Great stuff really , because otherwise I would miss it :)

[@loopingdope]

Primetime will upload a stream related to this

[@mitchellmnr]

@@loopingdope he is like a week late lol :D ...

[@ChrisAthanas]

There are other channels that deep dive it

[@EwanMarshall]

Yeah, it is worse than that though, the shorthand is wrong when it says sshd uses liblzma, it does not.

[@Wielorybkek]

this was probably the best explanation of the entire situation I've heard so far

[@Luxcy]

Did we loose fireship?

[@theEtch]

your honour, technically it was consensual as they implicitly accepted the license agreement before the penetration took place

[@ActionScripter]

a bot liked your comment so much it copied it ten minutes later

[@flannn6]

it wasnt the kernel btw. so it techinically wasnt linux

[@XDarkGreyX]

Had to akshually

[@MaZe741]

Real Linux has never been tried

[@nerdkartoffl9019]

Thanks for the camera analogy. Makes it easier to understand for a noob like me.

[@husninazer]

The analogy at the end was spot on!

[@mrtnsnp]

The even more insidious part is that sshd does _not_ use liblzma, instead liblzma is used by the systemd software (that starts and controls practically all other user processes on the affected systems). systemd also load the sshd software, and as soon as both liblzma and sshd get loaded into the same addressspace, the backdoor is activated. The hidden code replaces some internal functions within sshd, despite the fact that sshd does not depend on the library. I'm sure this will have repercussions both on the technical side, and on the people/trust side.

[@seeibe]

I don't think it's quite that simple. It seems to be done on the distro level to patch the ssh implementation, as for example on Arch linux this doesn't happen even if you use systemd. For the redhat distros it makes sense, since they also develop systemd, although I'm not quite sure why debian and ubuntu also do this.

[@mrtnsnp]

@@seeibe It is a complicated hack for sure, with many subtle aspects as to when the backdoor gets included or not. These include some fairly specific checks on the results of uname. As far as I understand it, the sshd code is uncompromised, as is the systemd code itself, the backdoor gets installed purely from the lzma library. What makes this backdoor possible is that systemd based systems load the ssh deamon into the same address space as liblzma. This allows the lzma initialisation code to replace some critical functions within the ssh deamon. I'm sure some design choices and availability of certain features within critical components will be reconsidered over the coming months. And do read Ken Thompsons "Reflections on Trusting Trust" (Turing award lecture in 1984).

[@angelorosa9534]

I think your analysis is quite wrong. As far as I understood , Linux systems does not entirely rely on xz library by default. Such library, has been used by some distros for sshd, to let sshd being able to display messages to the end user, which is done by systemd. Systemd does not interact directly with the compromised library unless being patched for displaying messages. The reason why arch is not affected.

[@galewallblanco8184]

that reminds me of that one attack that hooked into libc, and basically proxied all functions of it, as a consequence it would filter out itself from any standard library level function output, like... files, pids, twas insane

[@lucass8119]

To be clear, this wasn't a link against systemd, but rather systemd-notify. systemd is not a piece of software, its dozens of pieces of software. You also don't need to link anything to integrate with systemd - it talks to processes via signals and D-bus.

[@elimcfly350]

People like Andres make me realize that I'm not smart and, in fact, am actually incredibly stupid.

[@ad4m300]

Bro I don't want to sound cheesy but everyone has the ability to become smart/great at something. It's just depends on you if you are willing to put in the hard work and the hours. Never ever talk yourself down.

[@spinix3744]

It has nothing to do with you bro, disconnect yourself from whatever is going on. Andres is'nt always perfect, he just had a moment of ascension, there are times like that, where you connect with ultra cosmic consciousness and pay attention to detail and depth.

[@RolandoGarza]

@elimcfly350 or, you can conclude that a good person can make a lot of difference in the right moment.

[@o0Donuts0o]

Define smart? Do you think this guy can lay bricks to hold up the roof of a house? Or is it only a matter of learning the fundamentals that build knowledge?

[@elimcfly350]

I was mostly joking, fellas. I also didn't know that this dude is an engineer at Microsoft who was just doing part of his job, since this video never mentions that. I thought he was just a hobbiest running benchmarks for funsies. That's why I was thinking "dang, this dude is on a WHOLE other level of nerd."

[@nsa3679]

I'm addicted to Fireship. I need my weekly Fireship. Where is my Fireship.

[@kmmadhu1280]

I loveeeee the analogy you gave and it's absolutely SPOT ONNNN!!

[@beachbum868]

I'm so glad TempleOS is not affected. I have all my Productions systems written in HolyC.

[@lh8228]

cowsay bless you

[@MatheusKlSch]

everyone expects that open source projects are audited by peers regularly, but no one actually does so, because each peer thinks it's already audited by someone else and when people audit stuff, it's usually the end-user software (specially security software), not a vital but random utility library managed thanklessly by someone from the middle of nowhere suffering from burnout

[@furycorp]

Haha yep "anyone can review the code" doesn't mean that anyone actually has or is

[@boumajohn]

OSS Auditing is the Academic Reproducibility of the tech industry: was done in the past, is no longer done unless there is a big issue because the volume is too high and the code (experiments) too complex.

[@THER0RKEGUY]

Hey fireship, i just want to say thank you for the videos you make, big fan of the *this in 100 seconds* videos

[@mul555]

I lecture cyber security, so often have to relate things ti the "real world" and your non technical explanation is perfection.

[@MarquisDeSang]

Temple OS is our last refuge.

[@ispamalot]

always has been

[@klixikix]

@@ispamalot I was gonna post the same when I saw your comment rorschach

[@archuserbytheway]

It's the temple

[@themore-you-know]

When everything else fail, pray.

[@Stix_Zidinia]

CHRIST IS KING

[@user-cc8kb]

I appreciate that you added the US to the list of rogue states :D

[@JH-bb8in]

Fireship was too chicken to just say China 🇨🇳 and we all we know it’s

[@theali8oras274]

isnt a 'rogue' state whichever the US say it is?

[@armynyus9123]

@@JH-bb8in > we all know Falsified by one example: me.

[@Sunrise-d819i2]

yea, they been asking for backdoor for years. tho they are fail to own faults as most cyber attacks on USA was due to usa backdoors. water systems, power gird, and ect. A backdoor will always be used by your enemies or bad actors that finds it. its like a kid that cries each time he hits himself why he got hurt. If theirs a door it will always be used no matter how hard you hide it.

[@JH-bb8in]

@@armynyus9123 wasn't counting NPCs like you

[@SumriseHD]

I used to work for a company that got hacked and the only reason the employees noticed was the doors (that were controlled over the network) opening a second too late, because the hacker decided to copy all files he found without limiting how fast.

[@scibrilneom]

That analogy was great, allowed me to understand this more without knowing anything about coding

[@brunesi]

Not sure if I am more amazed by the injection code quality or by the fact and how it got caught.

[@primenumberbuster404]

This title is crazy bro _💀_

oil up bro be there at 9 🙏😭

[@1.4142]

1:01

[@RealRhythmandPoetry]

Do you think it was consensual?

[@user-qy1dy1ms9m]

Is that skull emoji slanted?

[@squarerootof2]

What's even more terrifying no lube was used. No one is safe these days.

[@Alt33347]

Don't worry he gone to take some milk

[@EFXTVe]

Quite a compelling narrative indeed! The intricate complexities of cybersecurity are starkly unveiled in this exposé. It's a stark reminder of the perpetual vigilance required to safeguard against such surreptitious incursions. Kudos to the elucidative presentation!

[@mon0theist_tv]

Imagine how the hacker feels. He was SO close, working for years, only to get busted right at the end. Put some respec on his name. If he'd succeeded it would've been legendary.

[@seeibe]

If the hacker is a state actor with a particular target using one of the rolling release distros, they may already have been successful. Who knows.

[@nanonkay5669]

This is the beauty of open-sourcing software. So many eyes are looking at the software that this "pure luck" will have a much higher probability of occurring.

[@Felipe3001miranda]

At same time is sad that only one guy is responsible for a major library that internet depends upon, and the only help he got was from a rogue agent...

[@ismbks]

good luck finding backdoors in closed source software, it's way harder to audit a black box, i guess security through obscurity is the real thing

[@JACKHARRINGTON]

@@ismbks Good luck patching them

[@Cassp0nk]

Some rando state actor wouldn’t just be able to contribute into closed source, so your point is really weak

[@nou712]

@@Cassp0nk Yeah, instead the american alphabet bois get to dictate to microsoft windows, solaris, unix where and how to put backdoors. Open source is your best bet at not getting backdoored by anyone. Closed source in current year is an almost guaranteed way to get backdoored.

[@teo-tsirpanis]

I learned of the concept of "source tarballs" some months back and immediately felt it was a bad idea. Source distributions must be the cloned repository, optionally with some files only removed, never added or changed.

[@Stonium]

An incredible analogy. Well done.

[@Lantalia]

Technically, sshd _doesn't_ use XZ Utils, except on a few distros that patch in systemd interactions. We got really lucky that the attacker borked performance of ssh logins, and that someone was profiling something that made them. How many attacks are in place that don't impact performance? How many side loaded dependencies have binary 'test' data and some obfuscated build logic? How many dependencies have burned out maintainers? This is all back to "Reflections on trusting trust". You don't need to compromise many systems with something like this, just manage to get onto the build servers and inject a broader vulnerability into binary packages without it ever being in those packages source control and the game is over.

[@piotrek7633]

This guy was like -Hmm?! HOW OUTRAGEOUS!!! How could this be?? CPU spike when i SSH into another machine? Thats extraordinary, never 'ave i seen this before! And the delay is a little higher than usual for this type of machine! Blasphemy, i ought to report this atrocity to the proper authorities! There has to be some kind of mischief happening!

[@Renovatio2142]

and there is me where i tolerate more than 3 years a wifi disconection for more than 15 seconds if you use more than 15 devices on wifi network...

[@leechys]

bring out the ping, get the top, last but not least ps aux

[@bernardonegri5416]

I remember he said that the CPU spiked and there was too much delay (500 ms) even when the username was wrong.

[@mchl_mumo]

Thanks for explaining it simply. I could see the info all over but didn't really get what it was about

[@netstereo]

Thanks Jeff, I hit the like button with so much pleasure. Love your editing style. Do you have a video about your workflow? What stock video service do you use?. Take care, Sir.

[@deltarno7502]

Thank you for mentioning that this is not a April 1 video. I was going to be going over this for hours trying to figure out the joke.

[@universaltoons]

TempleOS is truly the distribution of all time.

[@taahaseois.8898]

Well the thing is that it is not in fact a Linux distribution. Terry did his own thing.

[@FirephoenixX02]

HolyC FTW

[@ChillerStone2]

MISTAR BIST MRBREST IS THAT YOU ???? I NEED MONEY!!!

[@sensu31]

moistcritical looking ahh comment

[@seeibe]

@@taahaseois.8898 Yep. More importantly it doesn't have internet, because internet isn't necessary.

[@SwimmaaY]

i knew when i saw this all on twitter i could count on waiting for the video from Fireship to elaborate

[@smritips2510]

Maybe we are not sure about the hacker but the mofo whose cpu ran 500ms slower and he noticed it too is for sure a psychopath

[@dgSolidarity]

Yeah, it was widely reported he was a benchmarker.

[@trumpetpunk42]

3:20 "a rogue state like Russia, North Korea, or the United States" I always love fireship's subtle based takes! 😂

[@zekicay]

This is not entirely correct: openssh doesn't use liblzma, instead a patch to openssh-portable to implement systemd notifications loads libsystemd which loads liblzma.

[@reekdas9219]

best analogy ever, perfectly suits each detail.

[@Daimo83]

That analogy was amazing.

[@duckhuntergaming4713]

This is eerily similar to the event-stream npm package supply chain attack. I published a paper called "A systematic analysis of the event-stream incident". The first workshop we submitted to, rejected us, asking, among other things, to provide more countermeasures. The whole premise of the paper was that this attack vector is too subtle and difficult to detect, even by today's technology. We tried to warn the academic community, but currently I am not aware of a consistent way to reliably stop these attacks.

[@mikkelcornelius6948]

I literally switched to Linux Mint less than a week ago. If I understand correctly this wont bother me, but man seeing this title in my feed, made me blow air through my nose.

[@goncaloazevedo9822]

Personal computers are often behind NAT, so even if you were compromised it wouldn't really matter

[@MagnaP]

the analogy was really well put

[@CharlieAligaen]

I am just learning some cyber security. It's crazy to think how much we don't know all the exploits that are out there. Playing the long game on this shows how much patience these hackers have. I always remember Professor Messer's lessons to monitor the cpu usage.

[@justinholz480]

I was shitting bricks last night scrambling to recompile my router firmwares because I had built openwrt from source last week and used the main branch because the latest commit for yggdrasil v0.5 hadn't made it to the stable branch yet. Only to read the full report and realize it only affected x86_64 arch with systemd. Regardless OpenWRT released a commit rolling back xz to a reliable version so it was good practice verifying that nothing was compromised and rolling out the patch. Even if the backdoor had targeted the router architecture my systems would likely have been unaffected because ssh is not accessible over VPN or WAN, and Yggdrasil disables SSH access in the firewall by default. Phew...

[@falsemcnuggethope]

relatable

[@MrSpeedFrk]

On a side note, this same scenario happened to a Canadian company Nortel, it was pretty much entrenched in every major telco around the globe in the early - mid 90s Long story short , Nortel went BK, was one of the most epic flops and some of the engineers who worked at Nortel during the time now have their pictures displayed in one of the main areas of Huawei corp. headquarters as distinguished engineers As Nortel was selling off property , during renovations it was claimed that there were cameras found behind walls and other recording apparatuses , that information was soon "debunked" however

[@spacemeter3001]

That shit sounds ridiculous until you actually work for government contractors or major companies. Then this becomes a very real reality where you are not even allowed to put your own chargers into walls or they start using their own contractors for building renovations. JetBrains for example bought up some big apartment buildings in europe for their new headquarters and they did NOT use local renovation services but selected their own in order to prevent espionage 😅

[@magicmulder]

Excellent offline analogy. :) The real WTF is how the malicious coder got maintainers of other code to switch off a security check because it "has a false positive on my code". That should not have been done without massive code review. That was dumb on the same level as log4j executing parts of logfiles as code.

[@57thorns]

Love including USA in the possible attackers, because NSA is _not_ in the clear here. Meanwhile, at FOI, Sweden's defence research agency: "At least they did not figure out it was us, we need to be more careful the next time."

[@dekjet]

​@michaeljb3107NSA would put their backdoors where they can't be attributed back to them. They're more subtle than that.

[@username9774]

@michaeljb3107ever heard of the intel management enigine on all modern intel (and amd) cpus? NSA is already in the cpu

[@Yxcell]

Technically, sshd doesn't depend on xz/liblzma, but systemd does. Some Linux distros (but not all) made some customizations (patches) to the sshd in their package repositories so that it would depend on systemd for systemd-notify. The backdoored liblzma was able to detect if systemd was depending on it and if sshd was depending on systemd.

[@architech5940]

I would just like to say, I've watched this channel go from less than 100k subs to almost 3m subs in like 5 months span. Amazing.

[@vinaymodepalli4266]

Wow awesome channel. If you know any other channels like these which summarize all the info a developer need with short videos, please mention them in reply

[@RucadiDev]

Plot twist, imagine that the "mine" was planted by a security researcher just to get the prize for finding it