Unveiling the xz Utils Backdoor which deliberately opens our SSH connections for RCEs

In the latest liblzma update, a trusted bad actor called 'JiaT75' implemented a backdoor which allows RCE (sending calls to system()) on ssh connections. Here I'm looking into the case and explaining how it works.
Links:
- AndresFreundTec on Mastodon: mastodon.socia...
- openwall email: www.openwall.c...
- debian repo: salsa.debian.o...
- Filippo Valsorda on bsky: bsky.app/profi...

[@isbestlizard]

God bless we have open source developers CURIOUS enough to pull the thread of a 0.5 second ssh slowdown until it unravels, not merely to shrug and think 'huh computer must be acting a little slow today whatever'

[@dominikvonlavante6113]

An 0,5s slowdown is actually tremendous. But nobody pays attention to that detail.

[@bradleypout1820]

WOW amazing how you explained it, step by step BUT explained the little things average person wouldnt know. THANKS!

[@geekingjadi]

You're very welcome!

[@AllanSitte]

This should be a gut check to the open source community. Everyone should take time to review all components of their respective projects. Yes... everyone. Considering the complexity of this compromise (including the time spent to social engineer the project team), it plausible that this kind of attack may have been successful in other projects.

[@iilliya8]

for someone to go this length to put a backdoor,I don't think it be a person or even small groups of people this was definitely from a larger entity 😑

[@geekingjadi]

My blind guest is also a nation state level attack

[@iilliya8]

@@geekingjadi yeah I guess that so 🤔 lm thinking ,maybe there be other backdoors like this which still not been found! What’s your opinion?

[@tonyzone8999]

Ya our government. I’ve seen this attack done and I’ve also analyzed the code. It’s a government back door 100%

[@Batwam0]

@@iilliya8ssh is obviously a primary target to allow access but I hope that they will look around in case similar attacks have been included anywhere else

[@IMBlakeley]

Makes you wonder how many others there maybe

[@morningstar2219]

Thanks to YT Recommendation for showing this channel !

[@geekingjadi]

Thanks for coming

[@thanostzia]

Very nice video! Good explanation, was able to follow along even if I didn't have knowledge on some of the things that were mentioned

[@geekingjadi]

Glad it was helpful! And if you knew everything in the video, it was a waste of time to watch :) and if you grasped 100% of it in one go, you were not learning much. The best learning is when you learn some new things and have a glance on some other stuff which you may learn about in the future.

[@surfingbilly9654]

best explanation by far regarding this issue so far, keep it up!

[@geekingjadi]

Awesome, thank you!

[@k1tajfar714]

LOVED THIS!!! Super interesting. More and more videos like this please! Checking on the CVE s and stuff like that! To me it sounds more like I'm in a hackerish movie having the same journey with you. Pretty informative and entertaining video. Thanks.

[@obtron]

Wow, superb explanation. Subscribed!

[@geekingjadi]

Awesome, thank you!

[@StuXan0]

so basically he was using the infected xz library which was injected to ssh initialization method to literally login into every single server which is using key-based authentication on their ssh? tbh he was about to land a very historical attack, but fortunately he was unlucky enough to get caught fast

[@geekingjadi]

This was a remote code execution attack. And yes.. would be a historical access to all servers if found it's way to major distros unnoticed

[@danyalt8221]

Interesting and cool content, Thanks Jadi!👍

[@geekingjadi]

Glad you liked it!

[@forivall]

My guess with lang is that gettext lookups help obscure or enable the backdoor's behaviour. I don't know gettext, but I imagine it does some addressing magic to make translation lookups fast.

[@forivall]

Also, from what I read, xz is loaded because systemd uses it

[@geekingjadi]

Right right.. thanks for adding this. forgot to mention it. Will thumbs up so people will see it

[@LiEnby]

I dont understand how they made the tarball different to the upstream repo .. its not like he had access to github itself It looks like an autoconf rule to make it not include the backdoor rather than the tarballs actually differing?

[@geekingjadi]

I have not checked this personally because the xz repo is not public / accessible anymore. but consider these points: 1. github tarballs can be uploaded separately. So you can have code in your tarball which is not in your git. 2. The malicious code was in a "test" file and was summoned using a m4 file.

[@panagiotischagias4119]

Thanks for the explanation.

[@agnescode]

The video was exciting and interesting 👌🏼

[@geekingjadi]

Glad you enjoyed it!

[@tigros999]

but now some bad actors just learned some really neat tricks heh

[@vahidmostofi6940]

Thank you for explaining this. this was super exciting. Unrealted, but can you mention what Pen tool you are using? Hardware/Software

[@geekingjadi]

Sure thing! Its a super old bamboo from wacom. So old that the driver wont works on Mac anymore and needs patching. the software under mac is Screenbrush and under linux I use gromit-mpx

[@isbestlizard]

Is it possible to automatically diff every source tarball and the github contents and look for what could be extra lines in any of the other tens of thousands of packages being built?

[@geekingjadi]

yes. but it will be 1000s of lines of code. in this case, the distros are trusting the "trusted" packages... In some countries the state do have a "recommended" distro which is checked by auditors. But they should be using older versions of everything.

[@isbestlizard]

Is it possible to use AI to fingerprint the malicious script? Like, look at the way they use line breaks, spacing, indentation, quirks and ideosyncracies, to extract a fingerprint and use that to link to other scripts that use the same 'hand'?

[@johncarlson2632]

currently AI can’t reliably detect sometimes even basic patterns. It’s also very prone to giving false positives. Catching stuff like that is very high level digital forensics. Maybe a specific AI developed to recognize those patterns and trained to recognize those patterns could in a few years. But it would only be used as a compass and not a metal detector

[@geekingjadi]

AI works based on seen patterns / lots of samples. Your idea already works on things like "recognizing the vandalism on wikipedia" but we do not have enough/alot of samples in programming yet... so as @johncarlson2632 said, it is possible to train an AI, but there will be lots of false positives. And.... in this case, the person with high privileges has done this... he would disable / bypass the AI too.

[@Noname23489]

just curious.... why would large organizations be using operating systems that get updates from open source git repositories? maybe im understanding this attack wrong just curious if you dont mind educating me

[@geekingjadi]

Whats the other choice? Otherwise they have to use an operating system which they do not know anything about its source code / programmers / ... . If someone is doing something super critical, they have to have their own GNU/Linux distro and audit all the code they use. This means 1. a super minimal distro and 2. using the older software and 3. lots of expenses.

[@rar4-s5j]

🔥

[@Entropy67]

this should serve as a PSA, no binaries should ever be committed to open source, never in the codebase itself.

[@mechassistant]

Danke 😁

[@alimahdavi9775]

nice

[@miladganji1921]

انگلیسیت خیلی کیوته جادی 😍

[@lv3609]

It scape detection for almost 2 years, until some unique developer locked into the strange execution delay in milliseconds (most people wouldn’t care much). Meaning, this could have been going for much longer (if was not for that odd developer), bleeding heart hack is just another example. That is, millions of lines of code, trowing thousands or million of man looking at that millions lines of code doesn’t mean every bug or every hack will be detected.

[@NoONE-bk7ud]

you are just better

[@GyroO7]

so even code bros are not safe on the internet

[@AndreyDerkach8]

it's a huge lol

[@prajnaparamitahrdaya]

Anonymous

[@hqcart1]

the fact that repos are moderated by individuals makes the whole echo system unreliable.

[@geekingjadi]

Right. Another critical, single point of failure attack vector

[@Noname23489]

just curious.... why would large organizations be using operating systems that get updates from open source git repositories? maybe im understanding this attack wrong just curious if you dont mind educating me

[@hqcart1]

@@Noname23489 This is how life was going before, but i don't think it will be anymore, just like before and after 9/11, going to security checks has changed forever.

[@stefsot2]

stop repeating the same stuff 3-4 times with different words, it gets tiring

[@geekingjadi]

thanks for the hint. will try.

[@xamashee]

Best explainer so far 🎉 good on you

[@geekingjadi]

Glad it was helpful!

[@pedramardakani]

Tests and benchmarks save the day 😄This could add to the good reputation of tests and benchmarks :)) Great stuff Jadi. The content is great 👍

[@geekingjadi]

Very true!

[@AlexanderTrefz]

The exploit literally was hidden in the tests, this does not add to the reputation of tests.

[@pedramardakani]

@@AlexanderTrefz How come? It's not about the test files, environment, or functions, it's about the act of "testing" itself.

[@bowserlv100]

What I find most ABSURD about this is that practically NOBODY anywhere pays due attention to; HOW was it possible to see something like a failure where EVERYONE was blindly ignoring, TIME. The guy who discovered this flaw said: "I noticed an unusual delay in access... I ran a checking program and saw that there was an increase of 300 ms." NOBODY talks about it, no one cares and right now what must be backdoors are being hastily explored before they are blocked. FOR LOVE.

[@AndreaBorman]

Yes I have heard about this. I am a Linux user who changed to Linux two years ago after Microsoft stopped support for Windows 8. On Windows we would just install an anti virus program and scan our computer with that. We also had Windows Firewall. Maybe we will have to start using anti virus on Linux. There are anti virus programs for Linux as well as Firewalls. I am a bit surprised that no one has suggested this.

[@geekingjadi]

In most cases anti viruses are used on Linux to detect windows based viruses on share files, emails, ... . On Linux we trust our distro and the fact that no virus can not just spread to servers as it does in the windows world. Firewalls are also there, deep in the OS (iptables).

[@Entropy67]

lol on windows your antivirus is the virus, and the backdoors are built in to windows by some upset employee. Since no one can look at closed source code, the back doors go eternally undiscovered. Antivirus can't stop what it doesn't know, and it can't analyze that which has greater authority (the OS itself). The reason why you are hearing about this is because this is open source. You would not hear about it if it was closed source, you would just quietly get violated. After all, who would look into a half second delay on windows?

[@CrazyUncleJack]

The Linux version of “antivirus” is that it’s open source and the holes get patched quickly, thus utterly negating the virus. It’s a completely different paradigm.

[@atajahangiri5861]

VERY very GOOD video

[@Griimnak]

Great explanation Jadi

[@tajsec498]

خیال کردم از این ویژگی های جدیده یوتوبِ که جادی انگلیسی شده، بعد دیدم نه واقعا انگلیسی صحبت میکنه :)) کمی عجیب به نظر میاد. گریت ویدیو تنکس.

[@samas69420]

how do you draw on screen to take notes?

[@geekingjadi]

Screenbrush on Mac, gromit mpx on Linux

[@r.e.434]

what are you using for the writing?

[@geekingjadi]

on the screen you mean? screenbrush under Mac & gromit-mpx under Linux

[@md.hasanurrohmankhan5173]

Thank you 😊

[@geekingjadi]

You're welcome 😊

[@sepidehsoroush]

Very well explained👏🏻

[@geekingjadi]

Thank you 🙂

[@erfanffa]

great as always

[@geekingjadi]

Thank you! Cheers!

[@IMBlakeley]

The irony that the fellah that found this works for MS isn't lost on me.

[@geekingjadi]

right :) the irony.

[@a46475]

I heard from a certain internet personality that there is/was supposed an unprecedented worldwide cyber attack on the financial system this year and this would serve as the pretext to roll out a very different regime. For example requiring ID to go online among other things. I wonder....

[@glarynth]

The great thing about vague prophecy is that it can claim credit for predicting whatever random thing happens next.

[@a46475]

@@glarynth Unprecedented cyber attack on the global financial system some time in 2024 that ushers in identification requirements online among other things. Yeah I can see how that's vague and imprecise. That could mean anything. What degree of power would this entity have had, had this hack been successful? I doubt very much.

[@a46475]

@@glarynth Unprecedented cyber attack on the global financial system some time in 2024 that ushers in identification requirements online among other things. Yeah I can see how that's vague and imprecise. That could mean anything. What degree of power would this entity have had, had this hack been successful? I doubt very much.