revealing the features of the XZ backdoor

In this video we play with xzbot, the tool developed by @amlweems at Google, and use it to show off the scary functionality of the xz backdoor.
@amlweems: / amlweems
xzbot: github.com/amlweems/xzbot
original story: openwall.com/lists/oss-securi...
🏫 COURSES 🏫 Learn to code in C at lowlevel.academy
🛒 GREAT BOOKS FOR THE LOWEST LEVEL🛒
Blue Fox: Arm Assembly Internals and Reverse Engineering: amzn.to/4394t87
Practical Reverse Engineering: x86, x64, ARM, Windows Kernel, Reversing Tools, and Obfuscation : amzn.to/3C1z4sk
Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software : amzn.to/3C1daFy
The Ghidra Book: The Definitive Guide: amzn.to/3WC2Vkg
🔥🔥🔥 SOCIALS 🔥🔥🔥
Low Level Merch!: lowlevel.store/
Follow me on Twitter: / lowleveltweets
Follow me on Twitch: / lowlevellearning
Join me on Discord!: / discord

[@LowLevelLearning]

THANKS FOR WATCHING THIS WEEK HAS BEEN A ROLLERCOASTER ( come learn C at lowlevel.academy 🥺)

[@Star-uc8nt]

tycoon

[@Alfred-Neuman]

If I understand correctly, this hack wasn't discovered by auditing any source code, right? I heard the Microsoft guy just had a weird feeling about SSH executing his command for a few too many CPU cycles and was curious enough to investigate. Does anyone knows if it could've been easy to spot this by reading the source code or if it's hidden very well?

[@manobendro]

@LowLevelLearning Please revisit 2:36

[@joshkeegan3009]

Are there more videos to come on the topic?

[@moe3060]

is there a specific version of liblzma.so.5 or liblzma5.4.1 ?

[@Patashu]

'Binaries must be verifiably created using only the code in source control' would be a great place to start.

[@mitchellmnr]

The problem is this attack was done during the build process.... so you need the source to build it to verify it ... but its tainted ... so .... its valid? yea... this one was a wild one

[@khai96x]

@@mitchellmnr This is what the comment is talking about. The source of the build script must be part of the git repo.

[@mitchellmnr]

@@khai96x they are .... The way this attack works is when it runs tests on the source...

[@Zettymaster]

at that point, you should generate the binary on the fly. not that that would help when the build process itself is infected

[@jsrodman]

It's a decent idea but what do you mean in practice? Do you run your build in a way that refuses to give it access to any of the test data? That's not too crazy. Is it enough? Do you verify that others can produce your build without any of the test data? How do you ensure that the build process doesn't have a side channel running in both cases introducing other logic? What i'm saying is this idea has to be mapped to concrete practices and tools.

[@window.location]

Not enough people aware that maintainer/og auther (Lesse) is victim of social engineering, I read those messages in mailing list and it hurts me. Jia basically took advantage of his health condition.

[@LowLevelLearning]

yeah its really sad :(

[@TKing2724]

How's Jia doing these days?

[@channelgogrvk]

@@TKing2724we don't know that Jia is one person or an identity being used by a group of people coordinating the attack

[@eight-double-three]

​@@channelgogrvk I'd guess, it's the latter...

[@franzlyonheart4362]

@@eight-double-three NSA, Google, CIA … ?

[@dj1ch.]

we definitely need to see the investigation behind the contributor who created the backdoor. definitely very good at social engineering, being subtle, etc. if it weren't for Andres Freund we would have never been talking about this.

[@zirize]

We all know who it is, we just can't say who it is.

[@potato8236]

​@@zirizevoldemort?

[@ViciOuSKiddo]

@@zirize Three letter agency funded by a big brother?

[@saddish2816]

​@@zirizeit's not the NSA they have Intel management engine and AMD PSP

[@user-pk4hn1uz1k]

@@saddish2816 Why would they stop at a hardware backdoor? Ideally you want to control every step of the computer's stack from bare metal to the user

[@DavidRomigJr]

Jeez. Virtually every part of this is done in a way to be practically undetectable at every stage and effectively roots every bleeding edge version of the biggest distros and would have seeped into stable versions of them with time. These people are diabolical, and it’s practically a miracle it was found this early before it got worse. To think, detection started with someone noticing ssh connection speeds were a hair slower. Major kudos to them for being so prudent. And thanks for the updates on this.

[@draken5379]

Pretty much. I dont think it could of even been hidden any better. The main reason anything was looked into, is because someone noticed their SSH login taking half a second longer lol.

[@lucyinchat]

Except arch (due to how arch doesn’t patch sshd to connect to systemd-notificationd.)

[@skilz8098]

@@draken5379 Oh but it could be... What if your billion, no trillion $ chip - board makers are embedding hidden algorithms as such but as undetectable hardware roms that the OS, the user, etc. doesn't know about. Then with that only a handful of elite personnel coming from these giant corps as well as limited personnel within some government agencies have direct remote access into these hidden systems. Think of it this way; some of these newer systems you can load the entire Original Doom Source Code and Assets directly into Modern Cache and never have to go back out to RAM or to the Hard Drive. If you go back to the days of the sneakernet with the good ole floppy disks, people used to sneak malicious code into their 512 byte boot sectors. They can literally miniaturize a 6502, and 8086, or even a basic Pentium these days into a chip so small it would look like a little controller on the motherboard or even onto some peripheral card that one would have no idea what it is or was truly designed, intended for. This doesn't even account for modern UEFI systems compared to obsolete CMOS Bios type systems. And now with the advancement of A.I. and designated A.I. chips showing up... The sky's the limit. Well, skynet is the limit according to some... They can always hide it somewhere that is least expected. How about within a button on your polo shirt? You have to realize that we aren't just in the age of digital we are also in the early stages of Quantum Processing, A.I. Systems sure, but we are also on the stage of Nanotechnology. They can literally make programmable machines that are 1,000th the width of one strand of your hair. You can't even see them with your eyes. So if they truly want to hide something they surely can! And they can hide it just about almost anywhere they want. Even in the foods you eat and the liquids you drink! And this is not science fiction either!

[@johnmishell234]

His name is Andres Freund

[@Manavine]

...that feels like a paranoia syndrome waiting to happen because their fears have been proven so much more horrifically right than they thought. That's going to mark them, considering you already have to be pretty paranoid to check code over half a second. ​@@draken5379

[@johnfitzgerald2339]

8:56 Agreed that there are certainly vulnerabilities with open-source, but this also highlights the strengths of open-source: having a huge cohort of noble, intelligent, and vigilant coders to sniff-out this type of espionage.

[@arthurmoore9488]

It's frustrating because I deal with people every day who don't like Open Source, but have no problems using SolarWinds!

[@dascandy]

Imagine this happening with a closed source company. Somebody applies, silently checks this in with the binary files, and there's nobody outside of the company that even has a chance at catching these backdoors. How many of these backdoors already exist in commercial software? How many does the company owning the code know of?

[@justacat3639]

@@dascandy THIS. Every time i see somebody talking about how this is "an issue with open source", i think this exact thing. What if this happened in a closed source software? AT LEAST with open-source you're able to find out. With closed-source there's a good possibility that piece of code will never be looked at again because "it just works".

[@dascandy]

@@justacat3639Thinking back to some part of Windows NT4 shipping with a thing called "NSAKEY" (en.wikipedia.org/wiki/NSAKEY) and nobody knowing what it does or how it got there.

[@stokdam]

@@justacat3639It is the well known selection bias.

[@alexnoman1498]

If anyone's takeaway here is "I don't happen to use those versions of of that library, I'm good", you did not understand the issue. That maintainer is still struggeling. Other maintainers are still struggeling. Other libraries are still compromised. The Perp and their Patron are still out there doing this exact thing and more for the next 30 years together with lots of direct and indirect colleagues. The people introducing vulns are payed exorbitant amounts of money, because buying vulns is more expensive, less exclusive, while just as bad for security. Getting the maintainers of our critical infradtructure payed and having them audited should be our priority world-wide. Every company in every country uses FOSS, but vanishingly few pay for it. Ask your manager whether your company pays for the software they are using!

[@TotesCray]

While I agree with you in spirit, if you're running RHEL and paying for it, IBM isn't giving the maintainers of xz a cut of it, so just paying for FOSS in a corporate setting doesn't exactly solve the problem either.

[@Rudxain]

I totally agree. Corps generate hundreds of dollars per tool used, but they don't even donate a dollar. Then we get a core-js or faker.js situation and people lose their minds

[@wChris_]

whats scary is that this was only discovered, because the backdoor was slowing down ssh as well as generating a lot of errors in valgrind.

[@Aeduo]

To be fair, a lot of exploitable code generates errors in valgrind. It's good that there's that sort of rigor put in to release software at some point. Not a perfect catch-all, but certainly a point where some funny business could be caught. Or just to help catch truly accidental bugs in general.

[@scottschiller]

I'm sure I'm not the only one who was reminded of Clifford Stoll's "The Cuckoo's Egg" (1989), where a 75-cent computer usage accounting error ultimately revealed a remote attacker with superuser access.

[@Mallchad]

to be fair the attacker had to disable and ignore several address sanitization features and test checks. Hence why stuff like valgrind was already setup to catch stuff out for this particular project.

[@KingKoncorde]

so it does make me wonder how many of these backdoors are already out there, just not discovered?

[@fellipec]

And the attacker was working with one distro contributor to help fix the valgrind errors. And the said contributor, AFAIK, was helping in good faith.

[@FireStormOOO_]

I still can't fathom how incredibly lucky we are this got caught as quickly as it did. I still haven't found a system on stable that actually shipped the compromised versions. We don't always get a warning shot like this.

[@VonCarlsson]

The real question now is: how many times have we not gotten so lucky?

[@Noname23489]

just curious.... why would large organizations be using operating systems that get updates from open source git repositories? maybe im understanding this attack wrong just curious if you dont mind educating me

[@joshuavillwo]

@@Noname23489 A sizable portion of servers on the internet run some version of Linux. There's no good data, but I've seen stats that are well over 50%. And everything based on Linux is at least partially relying on open-source code.

[@brunomachtchaos7876]

/cries/ I use arch BTW :(

[@josephh6697]

@@Noname23489Everything in the world runs on open source. Every major company relies on open source for the foundations of everything. Every operating system, every router, every protocol, everything you know as "technology" is, at it's foundation, open source with few exceptions. Except windows, and that's only half true. The idea is simple - instead of thousands of compression libraries, we maintain a few in open source and everyone uses them. They'll be magnitudes better than anything in house because every major company, every hobby project, everything that needs that library uses it. Problems are solved once, for everyone, once. This means that some open-source projects have 40 years of optimizations behind them - you couldn't make anything better behind closed doors without spending millions. Apply that logic not just for compression - but for *everything*. Modern websites are 98% open source libraries, and 2% actual code. That's basically every modern website - built on the back of NPM. That runs on an open source webserver, that runs on an open source operating system, that sends data over routers using open source utilities using open source protocols made by Request-For-Comment public domain submissions. Welcome to the actual internet - where smart people made utopia without the stupidity of corporations or the meddling of shareholders. Linux is 98% of the internet. Yeah, 98%. Major companies use them because windows is well, shit. Linux can run everything from routers to servers, but Windows servers suck hard - because yeah, of course they would, they haven't benefited as much from decades of open source contributions.

[@MikeC1]

As much as I love the technical aspects of this, its not what we should be focusing on. The "nature of [this vulnerability] of open source software" was primarily and overwhelmingly social. The project had a solo maintainer that was hopelessly overwhelmed. He was conned by one or more anonymous individuals into accepting a malicious co-maintainer by both attacking his weaknesses as the project owner, and offering him the answer he was desperately looking for to save his labor of love project. He was pressured into trusting code and a developer without sufficiently scrutinizing either.

[@Dogeek]

My takeaway is that open source is only as strong as its weakest link. So many projects go to the graveyard because of the lack of maintainers, and there's a lack of maintainers because people need to eat, need a roof over their heads, so they work their 9-5 as software engineers, which is draining in of itself. So when a crucial piece of software has a solo maintainer, it's a prime target for attacks. If there was more compensations for open source maintainers, maybe some of them could afford to maintain the software full time, and as a result would not burn out and be less prone to social engineering attacks. Or we could just create a foundation for those crucial projects, so that there are always developers able to maintain the software, and need to reach a consensus before anything gets merged.

[@ZakBellinger]

@@Dogeek I think all the companies that used open source software in their commercial projects on a massive scale that command massive profits should be obligated to divert a fraction of a percentage towards maintaining these things. Or better yet when there's a massive fork and that fork goes on to make billions of dollars, leaving the original project in the dust and penniless there should be something that allows maintainers to continue doing what they love to do. I love open source but the (un)ethics of business undermines such a great ethos it really becomes disgusting. Sucks to see such openness taken advantage of again and again, and even more now since technology companies rule as the market leaders in today's world.

[@ichigo_nyanko]

Not even that, he knew the attacker for YEARS before he let him into the project. nearly 4 years. And the guy was nothing but nice and helpful until a month ago. Nobody, not a single person on this planet, would have not trusted him by that point.

[@SlinkyD]

​@@ichigo_nyankoI wouldn't trust him until I saw how he handled LE or drugs/alcohol. Being internet based, I don't trust nobody.

[@stxnw]

Many projects do not need active contributors. Just stop adding more features.

[@sirflimflam]

I feel really bad for Lasse. Jia probably looked like an angel sent from heaven to help with project maintenance at a time when they weren't doing so great. Then all this comes out, and they have to confront this nightmare with suspicion cast even on them. It's like a dad finally handing his keys to his daughter for the first time to drive on their own and within 15 minutes she drives through a crowd of pedestrians.

[@JamesTK]

Yeah even going through their commits the majority of them are benign improvements... And then this, with the seeds having been planted in the test files ages ago

[@demolish_united_nations]

More like a snake you sheltered finally showing its true color and biting you to the ground

[@ichigo_nyanko]

@@demolish_united_nations nah, the contributer, before the backdoor, was only ever helpful and nice. If you knew someone for years and they were only ever lovely to you - and it turned out they were a serial killer - it would be a horrible thing for people to imply it was only ever obvious (they were a snake, you let them in) when it wasn't.

[@atticus2274]

@@ichigo_nyankothe shit you nerds say lol.

[@sfall616]

​@@atticus2274 Nerd calling a Nerd a Nerd is peak irony.

[@4.0.4]

Remember kids: writing a backdoor is no excuse to write poorly performant code that wastes CPU cycles so much it alerts the authorities.

[@HolyGarbage]

I would not call this a vulnerability in open source, rather it's a strength. This could, and probably has, happen in closed source software too. They can just get some person to apply to a job and do the same thing. The only difference is that in closed source it's far fewer people that could potentially find it. I think one of the reasons it was found relatively quickly is precisely because it was open source.

[@antelope6826]

Agree 100% with this, the fact that open source code is evaluated and run by so many independent people is a strength that leads to things like this being found. If this change was to a closed-source tool it could sit in a repo for a decade and no one would notice :/

[@leexgx]

It was more luck and that the new version caused half a second delay that a person was doing performance tests at the time most other people probably wouldn't have noticed unless they was doing performance testing, this was pure luck as it wasn't that Mutiple people picked up on this it was just 1 person if they fixed the delay part it wouldn't have been noticed until a couple of years later on when some high profile targets got compromised and don't know how

[@eructationlyrique]

What's interesting is the timing. We see the attacker working hard to make Ubuntu include the compromised versions in their 24.04 beta, but also trying to convince fedora to include it in fedora 40. This makes me believe they might've been constrained by time at the end be cause these two releases are very important: The next Ubuntu is LTS and the next Fedora is likely what the next RHEL is going to be based on a year from now

[@damianlukowski9996]

systemd was transitioning to dlopen

[@ZeroUm_]

The program being built being able hook back to the build process and change the linker behavior, worse -during the test phase-, is also super crazy.

[@lonelybiscuit243]

It does some really complicated/shady things like altering the Global offset table and character recognition, I wonder if we ever get to find out who was behind this

[@MikkoRantalainen]

I think this is the modern variant of "Reflections on Trusting Trust" by Ken Thompson.

[@smnomad9276]

What's that and how is it relevant to the topic?

[@gabereiser]

@@smnomad9276you should read it. It’s about “how can you trust your own trust chain” and what can happen if that trust chain is broken. “You can not trust code you did not compile yourself”.

[@levonschaftin3676]

@@smnomad9276 go read about it, you have the entirety of the internet at your hands.

[@Acorn_Anomaly]

@@smnomad9276 A famous speech/presentation he gave as part of an award acceptance speech. It's basic premise is "what if I maliciously modified the compiler included with the UNIX system to do two things - 1) if compiling a system component, insert a backdoor that allows me full access, and 2) if compiling a compiler, modify the output compiler to perform these two steps." Basically saying, "how far do you trust what's been given to you?"

[@timewave02012]

Trusting Trust is a much deeper concept, and the reaction to it was the practice of Diverse Double Compiling. This trust gap is way easier to deal with, and more analogous to the use of nothing-up-my-sleeve numbers in cryptography. Distributing binary blobs alongside source code should be treated with great suspicion. Let test data be programmatically constructed, and therefore auditable the same as code, or by some other means be provably incapable of hiding machine code (e.g. pi).

[@ayanami-rei-san]

That level of obfuscation though, the backdoor glows very bright

[@lukarikid9001]

phosphorescent, even

[@GelatinousSSnake]

Stop being antisemitic

[@lionbnjmn]

@@GelatinousSSnake i thought glowing is a fed reference, what does it have to do with semitism

[@nou712]

@@GelatinousSSnake Glow = government agent, not anything relating to ethnicity.

[@Terszel]

@@GelatinousSSnake wtf 😂

[@fellypsantos_]

the scale of this attack is absolutely insane

[@adibemaxwell6111]

Oh? You mean how this backdoor didn't affect any LTS distributions used in servers or production environments but only affected rolling release and testing distributions? That scale? In other words, only desktop users who are stupid and naive were affected.

[@MacroAcc]

@@adibemaxwell6111 what are you talking about? the only ones that wouldn't have been affected would have been arch or other unpopular distros, if you used debian based distros (almost all), you were screwed. in other words, this is 9/11 of coding

[@jasonhamm3370]

@@adibemaxwell6111 My dude, what do you think 90% < of the internet runs on?

[@Aomicplane]

@@adibemaxwell6111 It targeted all Debian based distributions. The only reason only rolling distro's were affected is because that Freund dude just so happen to be benchmarking his PostgreSQL databases and therefore end up noticing the backdoor. If he didn't it could have been planted in LTS distro's as well after some time. Also, the bad actor has been committing to the repository for a few years and it's still being investigated if there's more malicious or at least sketchy code in there, which if there is it could mean that even older versions of the libraries could be compromised in some way, which would also mean compromised LTS releases. You have to be short sighted to think these sort of attacks is solely an issue for people running rolling release distro's...

[@LaughingOrange]

​@@adibemaxwell6111 If it hadn't been discovered, it would be on like 80% of servers within 6 months. The only reason it didn't is because some Microsoft employee noticed high CPU usage from sshd and looked into it.

[@m4rt_]

one way you could prevent something like this would be disallowing binary blobs, and if you need it for testing, etc, it should be generated by code that is readable. For example a shell script that generates a large amount of the letter a compressed with xz and then intentionally corrupts it. Though this doesn't prevent people from doing bad stuff like this, but it makes it harder to hide. (people can still intentionally introduce bugs in the code that can lead to something like heartbleed for example)

[@villesyrjala3354]

You could also mandate that test data must live in a separate repo, and must only be used while running the tests (in an isolated sandbox). Also the build system should be less flexible so that any weird stuff really stands out. But all of that is moot if it's the maintainer doing bad things and no one else actually reviews any of it.

[@markusklyver6277]

@@villesyrjala3354Also inspect binary blobs before they are uploaded to a repo.

[@johnmoore8599]

This is a variation on the theme. Open Source projects may only have one or two maintainers. We had a wakeup call with OpenSSL a decade ago. This is just a sneaky supply chain attack via compromise of liblzma and oss-fuzzer. Trust was assumed, but trust was broken. How will single maintainer OSS projects audit all commits to their project if they are hobbyists or part-timers? The payload was to be included in deb or rpm packages during build via checks. That was another way to ensure it got into the main distros, but was low key.

[@MrSmokinDragon]

Large developers that depends on smaller OSS projects needs to contribute to maintain all the projects they depend on. Either man-power or money.

[@ellielikesmath]

the main issue is the main issue that has plagued open source recently: business expects services and support from the open source community, but is unwilling to pay for them. in this case, we have a security critical library being maintained by a single person. in no sane world would i expect that to be secure. system critical libraries either need to be fully audited and then quarantined with no more updates possible, or they need well paid, dedicated, redundant staff and regular security audits. the new layoff culture and over investment in ai (and underinvestment in mental healthcare) is going to do massive, yet untold damage to computer security.

[@a_random_lemurian]

one of the games i play (openttd) uses xz for savegame compression, and now the latest release is delayed.. because of the backdoor

[@LowLevelLearning]

I LOVE openttd. didnt realize it uses xz D:

[@eight-double-three]

S**t just got serious - no-one is touching my OpenTTD! 🤪

[@wroomwroomboy123]

That's it!! I freaking knew I recognised xz from some ancient memory!

[@DVSoftware]

Not sure how OpenTTD would ever be affected, even if using xz with a backdoor, since it replaces RSA decryption function, and gets the payload from RSA key supplied to SSH.

[@a_random_lemurian]

@@DVSoftware OpenTTD may not be vulnerable to xz itself, but it certainly is vulnerable to GitHub taking down xz's repo and users getting pissed about potential malware being included with the game.

[@emilyweston460]

cs undergrad here. A lot of this is still over my head, but it is so interesting. Honestly just inspires me even more to learn. I appreciate your breakdowns that are very clearly presented and approachable.

[@whamer100]

what's really terrifying to me was how it was legitimately almost not noticed in the first place

[@Leonhart_93]

That's because everyone is under the impression that these kinds of things are rigurously checked by someone else, therefore no one checks them. This should be a wake up call.

[@TheOzumat]

Who would win: A multibillion dollar government agency, or one giganerd noticing his ssh takes 0.5 sec longer to log in? That's a W for the nerds, boys.

[@wroomwroomboy123]

@@TheOzumat Obsessive optimisation is the root of all evil - and that is why it also naturally finds it. Dude wanted to optimise performance and probably spent days untangling the mess he ran into - just because he wanted a near unnoticeable improvemet.

[@user-pk4hn1uz1k]

Everyone's calling it a W, but imagine all the projects that aren't being autistically benchmarked by someone with a unique problem.

[@leshommesdupilly]

The perfect, most complexe and most elaborate hack was avoided because one random guy found that one random program was a few milliseconds slower than usual 😂😂😂 Remember kids, always track the bloat lmao

[@ZombieLincoln666]

It wasn’t a few milliseconds, iirc it was 500ms

[@percy9228]

he didn't investigate because of slow down, "FWIW, I didn't actually start looking due to the 500ms - I started looking when I saw failing ssh logins (by the usual automated attempts trying random user/password combinations) using a substantial amount of CPU. Only after that I noticed the slower logins."

[@natam1616]

THATS IT! im done im building my own O.S. and all the libraries it need myself, and running on a custom board with a custom riscv processor

[@mineton1293]

Little do you know that riscv could also have a backdoor. mwahahahaha. You must create a completely custom architecture and custom silicon to be sure.

[@v0ldy54]

and also blackjack and hookers

[@natsuzkan]

@@mineton1293oh but then there's backdoors built into your circuit design tools as well, so have fun arranging logic gates entirely by hand

[@Arekadiusz]

​@@mineton1293 Easy AF, I'm taking my soldering iron and going to work :P

[@ingemarastrom4623]

Really thankful for Andres Freund's agile mind; finding a needle in the 'stack is hard enough; he found a bd even though he wasn't looking for one, anticipating one.

[@jhirn2957]

Plot twist. Andres has the private key. He did this all to rise to stardom in the security world. Being discovered only strengthens his reputation of supreme social engineer. He’s brilliant either way.

[@Sparks621]

It's cool that this vuln has been found, but the scariest thing to me personally is the very real possibility of similar backdoors that may already be present in other widely used OSS projects. The fact that even this one was found by total accident is genuinely terrifying.

[@benjaminpatrickpatrickgarr9291]

Small OSS projects like this usually have a very high bar for performance to meet, and they usually meet them no problem, so it makes sense that the valgrind performance anomalies were the reason why this one got discovered. Now imagine trying to discover "performance anomalies" in commonly used proprietary software...

[@GEfromNJ]

To me, the scariest part of software these days are all of those 3rd party libraries. You ever install something with "sudo apt-get install X", and you'll see a stream of things being downloaded. Even just one part of one of those libraries could be infected and who would know? MD5 hashes or whatever would do nothing here because this was compiled into the release itself. Frightening as hell

[@_devilfish303]

what do you think happens when you download an installer from say microsoft? the "scary" looking download text is obscured from you

[@jsrodman]

Dependency graphs give us the tools to fix problems once instead of over and over and over. They also allow us to be more effective and efficient in our work. They also let risks and failures cascade through our ecosystems in astonishing ways. Not all of the problems are even malicious. left-pad in nodejs was just someone unpublishing his work and it took down tons of sites. What's the answer? i have no idea. But if we focus on practical situations we can perhaps come up with practical mitigations that reduce harm, and improve.

[@christianbarnay2499]

@@_devilfish303 The point is Linux in its current state doesn't provide better protection than Windows. The slogan "you can always recompile everything locally to make sure the binary matches the source code" doesn't work anymore. Because every single Linux library is dependent on dozens of others. It's physically impossible to recompile everything on your own. And even if it was possible it would still give no guarantee as this attack was built directly with a hook to inject the binary malware during the compilation process without altering the source files. Yes Microsoft is not better. But it's not worse either. And this is the true issue here. Why bother with all the hassle of securing your linux if anyone can trash all your efforts with a single commit in some obscure source repo you never heard of?

[@goraxe01]

curl | sudo is the one that terrifies me. Literally random code from internet run as root. Multiple attack vectors there from ip / DNS hijacking through to just straight up blobs in the install script... Plus potentially no evidence of what you just downloaded

[@askeladden450]

​@@christianbarnay2499microsoft is not worse? If such a backdoor was installed in windows, it wouldnt get discovered EVER. Even a simple babckdoor wouldnt ever get discovered, which wouldnt even get merged if it was open source. Security isnt black and white. No system is ever completely secure. Its about reducing the attack vector. And the attack vector in open source or linux is extremely small compared to proprietary.

[@lis6502]

and thank you LLL for covering this step by step from the very beggining in a way that leaves no aspect untouched. Besides being entertaining to watch, it was also being very inspirational!

[@jfftck]

Isn’t the fact that this was caught proof that more people are watching what is going on in open source? This was the best case scenario for a malicious attack and the openness of the code allowed having all the evidence available. I agree that this should be the start of regular audits of crucial open source software. But the log4j exploit a few years ago might have had some influence on the Microsoft employee’s idea to dig deeper. I see this as a large opportunity for more secure code practices in open source.

[@ratchet1freak]

this was found by chance more than anything, it's only because someone tried to get a good baseline for their system that they noticed that sshd was doing more work than it should and dug into it If that didn't happen then this would never have been found

[@Cavi587]

Well yeah. But a lot of bad actors are now fear mongering so that people get scared of open source projects. We should doubt every project, for sure, but we have the ability to look at code and the whole build process and catch such exploits. But of course, as this example showed, there are simply too many projects for people to realistically screen with every commit. But we have to start being more cautious, starting with the most critical packages would be a good beginning.

[@helipad_huck1543]

⁠​⁠​⁠@@ratchet1freakBut on the other hand, if the library hadn't been open source, he wouldn't have been about to look into it as deeply as he did. He likely would've had to conclude that the developer introduced a bug, and at most reach out to them, which would've only served to tip the bad actor off that their backdoor was still too detectable. So overall, open source did "win" here.

[@Sandromatic]

​@@helipad_huck1543as it happens, someone else actually did exactly that from my understanding, and that was the patches that the second version of the backdoored version of xz fixed, some valgrind issues that were found by a maintainer of a different distro, so that's xz 5.6.1 :P

[@Pentium100MHz]

So, as I understand it, if the compromised version made it to the server distributions, the only way to guard against it would be to limit who can connect to ssh using iptables (or some other firewall), since the attacker would still need to connect to the ssh port to send the commands.

[@Sandromatic]

That, or install an uncompromised xz on that server instead.

[@benfinch4969]

Imagine how pissed off the exploiter(s) must be right now... "Noooo, it was perfect!"

[@WinterAyars]

They spent years setting all this up, too.

[@Uerdue]

And yet they managed to calmly disappear once it was discovered, instead of yelling insults at the xz mailing list. True professionals!

[@WinterAyars]

@@Uerdue As opposed to when they were setting this up where they (presumably this was them) were quite rude.

[@Cavi587]

I have a feeling that it's not just a simple uber haxor kid, this is too big and the setup is too perfect in the way it was being set up slowly. I don't want to sound like a conspiracy theorist but I just have a feeling that the guy who we all saw as a co-maintainer is not the sole brain behind this attack. It had to be a group who would greatly benefit from having this backdoor available.

[@Uerdue]

@@Cavi587I think most people consider a state-level actor (like some intelligence agency) the most likely behind this. It's not just the amount of time/work spent on it, it's also the likelihood of failing which was too high for the average cyber criminal without government funding to risk so many resources on it. What if systemd drops its xz dependency 2 years into the project? Or another contributor shows up who likes to do his own improvements to the tests or build system (that touches your manipulated files, so you have to turn down all of their PRs for made-up reasons)? Or the original maintainer is just stubborn and says "no binary blobs of unknown source as test files, period"? Or some person simply finds the backdoor before it makes its way to stable distros? There are so many ways for such an operation to go wrong. You either have to be able to scale it, working on a bunch of these in parallel and hoping for one to succeed - or you have to just be ok with the operation failing and the only thing gained from it being the insight that it's quite a hard thing to pull off successfully.

[@acemany]

this backdoor is just... brilliant

[@7DuRd3n]

how do people come up with such ideas

[@rawallon]

@@7DuRd3n In the shower

[@rian0xFFF]

I've heard something like that, I thought it was common

[@percy9228]

@@7DuRd3n geniuses that have specific domain knowledge sharing ideas with others in their domain knowledge. this most likely was a state operation. you think this is wild, imagine the concepts and ideas in modern mathematics where you are not constrained by physical limitation. everything is abstracted and abstracted.

[@manzenshaaegis8783]

One key correction here is that the hosted git repo does not contain the necessary code to build the backdoor. Only the private repo for the library maintainers contains the necessary code. Your highest upvoted comment already starts off on the wrong track. I spent a lot of time sorting through this confusion today.

[@oliverhuth5662]

Wow this guy is some sort of evil genius. Appreciate the constant video updates, it's great to have someone summarise these really complex topics but still keep the essence of whats happening.

[@franzlyonheart4362]

Guy? Or guys, plural? Or … state sponsored? NSA, CIA, ((Washington)), etc.?

[@marblecar1162]

@@franzlyonheart4362 its always a government conspiracy, eh?

[@MechazoidApocalypse]

@@marblecar1162 It does make sense in this case though

[@johnmishell234]

@@marblecar1162For you it's never one

[@ancientaccount7982]

@franzlyonheart4362 why is everyone in these comment sections assuming it's got to be an American agency? My money would be on China, Russia, or Mossad. Hell, Mossad's bread and butter is this kind of diabolical social-engineering into unnoticeable attack vector.

[@LarsFL]

I'd love a video that summorizes the entire thing in one, to be able to send that to people and inform them

[@Zuriki09]

Binary blobs are basically immune from standard audit, I don't see much of a way to prevent this kind of attack except to ban blobs from being included in source control. Which is obviously impractical in certain scenarios.

[@MyCodingDiarie]

I've been struggling with this topic, but your video cleared it up for me. Thanks a ton!

[@Fudmottin]

I don't think the brilliance of this evil attack can be understated. Not only is it a backdoor, it is a backdoor where only one person or group has the key to open!

[@ScottHess]

An issue with this entire thing is that we keep trying to invoke a technological “Web of Trust” to let us bind dozens of dependencies into a single binary, but instead we should perhaps be encouraging larger teams to form. Many projects have one or two or three primary authors who carry the entire load alone, which is intrinsically broken over the long term.

[@johnathanstephenson8107]

Can we agree that whoever made this backdoor was pretty skilled Also the guy who found it is a hero for being diligent and trying to optimize and finding it

[@elkvis]

A couple thoughts on prevention. Move the test environment to a temp folder, completely separate from the source, and only run the tests on a stripped binary, so that the actual source cannot be found by the tests.

[@thegame4027]

Everyone will forgett about this in a month and we will be back to pressuring maintainers for open source projects to work harder and faster in no time.

[@AnalyticMinded]

I saw the twitch livestream where he followed the command cmd with a shutdown... and then his vm went dark. That was so awesome.

[@mustafa_el-rashied]

a great father who spend time with his children AND keep up with youtube, keep up the great work!

[@mohamedmonem2645]

This whole thing glows brighter than the sun

[@dylanoonk6456]

the NSA must be pissed that they lost their best backdoor

[@outtakontroll3334]

more likely the chinese

[@anatolydyatlov963]

As if that was the only attempt - it's the only one that failed, but numerous others have succeeded

[@bryede]

Yeah, this isn't a lone actor. It shows you how much more complicated it is to compromise an open source project.

[@poisonouspotato1]

@@outtakontroll3334 both

[@anirudhg5629]

Would the NSA really be silly enough to try and backdoor an open-source project?

[@TweakMDS]

Great demo. From a cybersecurity point of view, this should teach you to think bigger, implement zero trust and secure your platforms beyond just the front door. Treat every server/service as if it's already been compromised, and limit access from that perspective.

[@kapytanhook]

Just so impractical :( I just want to use my system and write things that jave some power to alter things. A big stack of sandboxes and passwords is so frustrating

[@hellowill]

@@kapytanhook then we need to make that easier. I hated the idea of 100s of passwords (for example), but a password manager has made that easy.

[@kapytanhook]

@@hellowill yeah, honestly. But this is deeper, it's that you can't trust your own machine or things running on it. People put backdoors in everything. We need well funded open source hardware and software but we are moving away from that

[@t0rg3]

People should be sifting through all kinds of archived conversations involving project maintainers looking for signs of social engineering.

[@misarthim6538]

This is crazy. But I think this just shows the power of the open source. This would be completely undetectable in closed source software. Also, mass audit needs to be conducted, the level of sophistication of this attack makes it almost certain this wasn't the first time they did this. God only knows what has already been compromised.

[@BillAnt]

All thanks to a bored and curious Microsoft engineer who was running a timing test on the SSH login process. The time difference between a normal and infected login was less than 500 milliseconds , and that's what tipped him off. It's pretty remarkable.

[@misarthim6538]

@@BillAnt That's a story of human evolution haha.

[@danielbaulig]

I've been following this as it was unfolding over the weekend. Just watched the first 3 minutes of your video. The fact that this backdoor is very effectively secured against being used by anyone else than the attacker is a super strong signal for me that this is not some random cyber criminal but a state actor. The sophistication and attention to detail in this whole exploit reminds me a lot of stuxnet. Honestly, it's a whole notch up from stuxnet in a lot of ways.

[@somejoe7777]

Without question. This was a well-funded state-level attack, likely one of the 1st-world governments.

[@outofbound28]

So, are you using a virtual machine setup on an air gap? I usually use a set of virtualbox machines in their own virtual network on an old notebook for stuff like this. Would love to see how your setup looks for cases like these.

[@maurocamerini90]

I finally understood his with your video, thank you!

[@MattRobinsonDev]

Really great breakdown!

[@fuyukaidesu1641]

But how would the attacker know which server got infected? Does it phone home in any way, or was the attacker hoping to just attempt connecting to random open ssh servers accessible through the internet by scanning IP ranges?

[@aeghohloechu5022]

the latter is very viable, given that there are botnets pinging every single public IP address right now in hopes of hitting something

[@gie4830]

Attacker prob patiently waiting for their code to be published with stable release then do crazy stuff. Remember it got exposed just because of 1 dude noticed ~500ms delay on his ssh connection.

[@gie4830]

After the hard part is done, the attacker just hopes the biggest targets are infected.

[@oleksandr8482]

@@gie4830 500ms is crazy delay, no way this backdoor will come to stable

[@fuyukaidesu1641]

@@gie4830 I don't know. I'm pretty sure actual big actors don't have SSH servers directly addressable through the public internet, you most likely need to jump through some enterprise VPNs to get access. So the targets were mid-sized. At least now he's vulnerable to getting honeypot'd. Wonder if someone will attempt to catch him like that. Though it would have been easier to not alert everyone if he really was targeting only a few actors.

[@Uerdue]

In cryptography, there is this concept of using a "nothing-up-my-sleeve-number" for (technically artbitrary) seeds / initial values within a cryptographic algorithm. For example, the Blowfish cipher uses digits of Pi as initial values in its key scheduling algorithm. This is done to prove that whoever invented the algorithm did not backdoor it by picking constants that are vulnerable to some sort of cryptographic attack only known to him. Maybe the same concept should be established for binary test files. Use known, publicly available images that rarely change (such as the Wikipedia logo). Or, even better, use a well-known, deterministic pseudo-random number generator seeded with a fix "nothing-up-my-sleeve-number" to generate your test files. If your test file needs a specific structure (that is e. g. known to not suit your compression algorithm well), understand what is needed for the test file to behave the way it should, formalize that and then again generate files of the needed structure in an accountable way. This would make it practically impossible to hide stuff in these files. (And additionally make the repo smaller in most cases.)

[@alexnoman1498]

You can find any arbitrary sequence of digits inside of Pi if you search long enough. Meaning you can find a sequence you like because it's vulnerable, then pretend like you rolled a dice.

[@thewhitefalcon8539]

That's just silly. They're test files.

[@Uerdue]

@@alexnoman1498 Indeed, which is why you would want to use digits starting from the very beginning. (I think the Wikipedia article on "nothing-up-my-sleeve-numbers" also notes this in one of the first paragraphs.) I just left that part out in the initial comment because it was already long enough. Anyway, my point is: Binary (test) files of which you cannot explain how you created them are inherently suspicious. They may contain a rickroll link stored in low order bits of some pixel values, the mighty flag if you're playing a CTF challenge, or a secret backdoor to-be-extracted by the build script, as in this case.

[@scaredyfish]

The starting point would be its own magic number which should be a 'nothing-up-my-sleeve-number'@@alexnoman1498

[@Turalcar]

That concept exists because of stuff like Dual_EC_DRBG (although Blowfish predates it).

[@TechnopolisDotTV]

7:20 too late buddy. I will WinNuke95 you! 😎

[@LowLevelLearning]

LOIC INCOMING

[@TotesCray]

Enjoy your new cupholder *ejects CDROM drive*

[@byranjohnson5931]

Great video! Thank you.

[@LowLevelLearning]

You are welcome!

[@realbatman.official]

well anticipated video

[@x-user3462]

"turn yourself in" xD

[@hsmiranda]

Great video! Thanks for information.

[@ryansteiger6960]

Thank you for the video 👍

[@foobarf8766]

Sheer luck it didn't get into the next Ubuntu, attacker missed the change freeze by hours, this could have been massive 😳

[@_mmarciano]

Super valuable work! Outside the tech community no one really cares, but this stuff is HUGE. Thank you for what you are doing

[@MohamedAnouarKhediri]

Great content 🎉 thanks for sharing

[@LowLevelLearning]

Thanks for watching

[@ExaltedVanguard]

This slipped through the cracks because it utilized binary files used as "known good" (or bad) values in tests. A fairly straightforward way of reducing/eliminating this attack surface is to instead use a hash of the generated data instead of comparing it to a "known good" binary.

[@ANONAAAAAAAAA]

We should value the simplicity of software a lot more to reduce the attack surface. I would like to appreciate simple software even if they are a little bit underperforming and feature poor.

[@ANONAAAAAAAAA]

Also, code freezing or "don't fix what isn't broken" was not so bad idea after all.

[@DaveBucklin]

Perhaps the popularity of BSD is about to increase!

[@diadetediotedio6918]

This has nothing to do with "simplicity of software" but whatever

[@diadetediotedio6918]

@@DaveBucklin I hope, BSD is such a great system

[@ABHISHEKJAIN-wv7bh]

This is absolutely wild social engineering attack, jia tan enters fix a non problematic bug which decreases efficiency of system (which has negligible contribution before and after)and then jigar kumar (which has no activity before and after) pressurized developer Lasse Collins to maintian the code , they found the weak exploitable link of the chain Lasse Collins, then Dennis Ens enters and pressurizes even more led Lasse collins to forcefully pass the baton to jia tan and the actors played well and jigar kumar forces jia tan now and then they collectively triggered a hard to trace a backdoor but putting a corrupted code, this is insane because i studied cryptography and algorithms to encrypt which are generally very hard to detect i fear that they can be successfull in what they want to do ,they are really smart guys targeted and 15 year old weak link ,imagine how much backdoors does open source and god knows how much in close source software there is a urgent need for "V for Vendetta "for developers, we need to protect developers like Lasse Collins these guys are pillars of human evolution literally.

[@0dayCTF]

This is so wild, excited to see what else is found!

[@DominicDW8]

Terrifyingly well executed 🥶

[@fritzw]

I don‘t understand how people can frame this as „a problem of open source / linux“. It is only BECAUSE the whole thing is open source, that this backdoor could be found AT ALL. Think about how it would be infinitely easier to do this in a closed source software!

[@MrSmokinDragon]

not necessarily. Many OSS projects have only single maintainers, which makes this kind of attack easier. Large software developers have extensive code reviews with many eyes looking over changes before commit, meaning that a malicious actor (other than the company itself) would have to fool or corrupt many developers to sneak in the backdoor. I am not saying that Closed-Source is in any ways better, what I am saying is please don't try and make it seem that OSS is always better able to withstand these attackers. OSS is much better on average than closed source, but lets not get complacent, but try and figure out processes to prevent something like this in OSS in the future.

[@fritzw]

@@MrSmokinDragon I agree, that this wakeup call should encourage everyone to be more on the lookout for such things. My point is, that you can ONLY do that in open source software. With closed source software, you never know. Sure, large companies also do code review, but not all widely used software is written by large companies with good practices in place. And good practice sometimes get thrown under the bus when time or money is short. If you can't see the git repo, the build scripts, the test files, then you have no way find out why your your program is suddenly slower by a factor of 4, or if some malicious employee committed some questionable binary blobs some time in the past. *shrugs*

[@bryede]

@@MrSmokinDragon Yes, but the makers of closed software can put these things in on purpose with no one knowing. We know governments are the biggest cartels in the world and can get your cooperation.

[@goraxe01]

Heard about the nist recommend crypto standards.... Quite a few have mysterious issues. Check out en.m.wikipedia.org/wiki/Nothing-up-my-sleeve_number (the counter examples section). Why subvert the code when you can subvert the committee defining the crypto algorithms everyone has to implement. Espionage predates computers.

[@draken5379]

No lol ? How exactly are you going to get your code, anywhere near, lets say, KZbin or Photoshop`s source code ? I mean sure, you could try get a job at said place, work your way up, and etc etc. But that is vastly more work/time and secure than letting people who created a github account a week ago to contribute to key aspects of the worlds infrastructure. Like jesus.

[@valcaron]

Probably a good idea for Linux community to do a massive audit of all kernel and userbase tools for all other instances of "developer nobody has heard of pops out of nowhere, commits source to the project, then disappears into the fog".

[@Kyuubi840]

It's much worse than that. Jia was a contributor to the xz project for 2 years, and prepared the backdoor through that time.

[@cherubin7th]

People reviewed the code, but the backdoor was not inside the code.

[@thewhitefalcon8539]

But that's, like, every patch.

[@goraxe01]

And openssh comes from the openbsd community. I was thinking of attempting to construct a graph dependcies for anything running as root and/or network accessible. Build some scores for the source repo (number of commiters, frequency of releases, cve reports, static analysis, recent binary blobs added) combine that into some form of help / funding needed dashboard

[@Uerdue]

As I recall, Linus said in an interview that about 50% of commits to the kernel are by one-timers that never come back.

[@fazlerabbi2666]

i don't know enough about the actual impact but this feels like the most of the internet could've gone dark due to this

[@kavehtehrani]

Love your videos!

[@Veilzlol]

And now you have to wonder what is out there that has not been discovered by 500ms noticers

[@trudyandgeorge]

ARE YOU KIDDING ME?! This was discovered more or less by accident. How unlucky is the perp. He would have had a golden ticket to some serious wealth if he got away with it. And how lucky are we?! This vuln blows my mind.

[@allezvenga7617]

Thanks for your sharing

[@ArtemYakovlev]

Thanks for another great video

[@RonnieRedd]

Starting to sound like a government 3 letter agency at work in the night 😮

[@Uerdue]

*GCHQ hackers sighing with relief*

[@phoneix24886]

This is MojoJojo level Evil.

[@guitarbuddha74]

wow I was thinking it still would have ran as an unprivileged user since I figured they were just getting login but it makes sense it would be able to execute as root since they are using the daemon process to execute. This was a really informative video.

[@Gunzy83]

Imagine if the community took as much interest in the original package as they are in reverse engineering this.

[@wallyhall]

The unix philosophy of “many things which each do one thing really well” could make attacks like these harder. SSHD does a lot - and it’s inevitably network facing. Breaking it down to have a separate (unprivileged) process which handles everything up and unto authentication and setuid… wouldn’t necessarily have *prevented* this, but would have made it a lot harder to pull off. I’ve always been a little surprised SSHD has been this way for so long - given most other services make every effort to run unprivileged, then offload switching user to some other process.

[@JMurph2015]

I mean sshd can already hand off to libpam for authorization iirc, and realistically, authenticating requests and launching a shell is kinda the whole point of sshd.

[@nurmr]

sshd _does_ support privilege separation though. I assume the backdoor activates on the secure side of it though (or disables it entirely).

[@n1t_]

Finally i just finished your video with Prime

[@markosdelaportas3089]

Had libzma not been open source we ought to forget about ever discovering such implement!

[@wnbdriver]

and here I am thinking my highschool era "backdoor" which was just a DNS server that opened port 22 with iptables for me whenever I queried it with a magic string was clever :D

[@xeqqail3546]

Can you rickroll everyone with this?

[@SuperMixedd]

We definitely need TempleOS

[@Treviath]

The thing that people need to start thinking about is "How do open source developers get paid". Furthermore big companies need to start spending on projects they rely on.

[@luchomcmenu564]

This backdoor is really moving my career forward

[@qexat]

should be said that ldd shouldn't be run on untrusted binaries :P

[@SuperMixedd]

can i haz ldd on macos

[@russianmadman322]

Signature algorithm does not always equal an encryption algorithm. What's been said in this video applies to RSA, but doesn't apply to ECDSA, for example.

[@klauspsch]

The public key is used to ENCRYPT and the private to DECRYPT.

[@CheCheDaWaff]

He was talking about signing not regular encryption. He got it the right way round!

[@tablettablete186]

Not for signatures, he is right.

[@scbtripwire]

Wow, this is nuts! 😮

[@MrVecheater]

1:54 "thank god the researcher at Microsoft found it" Andreas Freund: 🙂

[@gareth2021]

andres freund is legit a hero

[@IDJEGOI]

You could say he is a freund to us all

[@Mr.Carrot]

"don't leak my IP adress plz" Lmfao

[@tielessin]

Makes you think how many of these backdoors are around that have NOT been found. Scary.