What Everyone Missed About The Linux Hack

The xz exploit pushed the limits of social engineering, code obfuscation, package distribution and more. I'm concerned the important parts aren't being covered, so I decided to do a vid
FOLLOW LOW LEVEL: @LowLevelLearning
This blog post carried the video:
robmensching.com/blog/posts/2...
Follow Rob as well: / robmen
Maintainer's blog post: tukaani.org/xz-backdoor/
Diagram: / 1774342248437813525
S/O Ph4se0n3 for the awesome edit 🙏

[@meschine]

Thanks for highlighting this topic, Theo. We need to do more to support OSS maintainers. I share your feelings of anger and horror for this maintainer: Lasse Collin. While writing my thoughts down, I tried to hard to keep _most_ of the anger out of the text but my keyboard suffered. This is a particularly scary situation but I worry because its not uncommon. It needs to change.

[@t3dotgg]

Thank you for helping me realize and share the importance of the open source maintainer experience. Incredibly thankful for your post. Hopeful that my frustrations are at least a bit cathartic 😅 My heart goes out to Lasse. I hope this is the start of some real change.

[@ChuckNorris-lf6vo]

@@t3dotgg If the low level dudes need so much time to dissect this then this has been quite expensive to make so someone paid good money for this exploit. Like that hacker tv series lol.

[@emoryotott2055]

More of the companies that build (collectively) trillions in value off of those OSS repos need to fork up and pay maintainers

[@edwardcullen1739]

I think "Get forked!" should become the standard response to any individual being rude and aggressive toward an OSS maintainer...

[@tsr-vq8yn]

@@ChuckNorris-lf6vo "so someone paid good money for this exploit"... Yeah, maybe. "Whether it be the intrusion of hackers, a major explosion at the World Trade Center, or a bombing attack by Bin Laden, all of these greatly exceed the frequency of bandwidth understood by the American military. The American military is naturally inadequately prepared to deal with this type of enemy psychologically, in terms or measure, and especially as regard military thinking and the methods of operation derived from this." - Qiao Liang, Unrestricted Warfare p.344 - 45 | Published in 1999

[@FabianLopez_lomba]

Imagine finding this exploit only to be called "a random Microsoft engineer"

[@ark_knight]

i honestly feel like its such a disservice to him, having Microsoft's name called out everytime anyone speaks of him. I really don't feel his discovery had anything to do with the work he does for MS, but simply because he got curious about the high CPU usage. MS should get absolutely no credit for this. Just credit for banning 1 repo and 2 github accounts, one of which is being heavily debated by the community.

[@DuRoehre90210]

@@ark_knight Wait, why not? If they are paying his bills by the end of the day then they are actually doing good to the FOSS world. Just imagine that guy to have a less well paid job, so he would have to do more jobs and not have enough time to investigate on such issues.

[@ark_knight]

@@DuRoehre90210 That's your head canon. Any other company would be paying this man if not MS. He clearly knows his stuff and what he is doing. There is absolutely no reason MS gets credit here. Perhaps you like letting faceless corpos take credit for your work.

[@jceggbert5]

​@@ark_knightbecause "the competition accidentally fixed it" gets clicks

[@ark_knight]

@@jceggbert5 tbf, only a very few are using that as clickbait and are genuinely referring to the sage who discovered this as guy from MS. I understand nobody (or most) actually mean nothing by it, but MS is getting credit either ways as more people say it. Edit - it sounds like I am more opposed to this because its MS, perhaps that is my bias. But our "Friend" (haha, get it?) should get more credit.

[@DarylMetzler]

This attack hit the entire software exploit playbook. Built trust? Check. Socially engineered a situation? Check. Built an elaborate, difficult to detect exploit? Check. Managed to infiltrate a wide scope of possible downstream systems? CHECK! I hope there is recourse against this (these?!) bad actor(s).

[@bible1944]

this attack is absolutely insane (it's these btw)

[@asdfghyter]

@@bible1944 difficult to say, since it could be sockpuppets, but it seems too sophisticated to just be the work of a single person

[@raul0ca]

To think that this is the only package the threat actors had infiltrated is...naive

[@blarghblargh]

@@raul0ca with a certain level of paranoia, you can't run anything but silicon your company made inside your own factories in your own country

[@Eagledelta3]

@@blarghblargh Only with people you have known since birth

[@planetmarshalluk]

Really interesting video, I do think that the developer who discovered the exploit should be given a bit more respect than just "some random guy at Microsoft". They clearly went to a lot of effort and care about the quality of their work.

[@t3dotgg]

Very fair point, my mistake on that. I planned on going more into him when I recorded the intro and then got distracted by the social engineering stuff 🙃

[@planetmarshalluk]

@@t3dotgg the insights into the social engineering stuff is really valuable, and it definitely makes me think more about my online interactions.

[@retro8263]

that was nsa/M$ job

[@pdoherty926]

I have a feeling Andres Freund will get the credit he deserves in the long term, a la Marcus Hutchins. However, I agree he's been a footnote in most coverage and there's been no shortage of people poking fun, calling him a nerd, etc. on social media.

[@Flamin362]

@@pdoherty926 Andres surname is very fitting for this attack, but it might be unfair to name it after him... Jia Tan was pretending to be a "Fruend" to the project

[@spicybaguette7706]

*the biggest _discovered_ exploit Who knows what's out there... genuinely scary stuff

[@mstech-gamingandmore1827]

I fear this already happened. Someone _needs_ to investigate the core open source projects everyone depends upon

[@asgacc8789]

Even from Jia alone. They have been a maintainer for 2 years, what if this is not their first exploit.. I use they because Jia could be a team of bad actors behind the acc

[@frydegz]

In the end, the git history is clear. The only way this happened this time is because the maintainer snuck it into the build system and no one caught it, but before had hidden this data into test files. It is not impossible for those test files to have been decompressed and caught. Which makes OSS a good thing for this. When was the last time you decompressed your coworker's binary test files to check for backdoors? People are much more paranoid in open source, and that's why this was eventually found.

[@GreatTaiwan]

@@mstech-gamingandmore1827 why trust that someone

[@MacGuffin1]

Lol, this isn't even close to the biggest know... biggest this month maybe

[@Lucas-gt8en]

Dude this poor original maintainer. Even when you somehow ignore the chaos and felt responsibility there’s also the fact that somebody that he trusted lied about probably pretty much everything. I’d be genuinely surprised if this was not orchestrated by a state agency of a major country (US, Russia, China, Western EU) but I doubt we’ll ever find out

[@Omnifarious0]

My assumption upon learning how carefully the attack was carried out and how long the person maintained focus on it, was that it was the work of a nation state. So, I would be surprised if it wasn't.

[@sub-harmonik]

@@Omnifarious0 depends how much time the bad actor spent.. many maintainers work full-time as well, so it could be someone just looking for stuff to sell on the dark web or something.

[@GreatTaiwan]

@@sub-harmonik u can make waaay more money and waaay faster than build a biinary backdoor in commits for months and become a maintaner after 2+ years he could have done literely anything and he will make waaaay more money this is not financially motivated no way

[@Dratchev241]

read a blog yesterday about this, they broke down the name dude used and according to the person breaking the name down its a fake name to appear as a Chinese person (something about the first/last isn't a name you would see due to some Cantonese vs mandarin naming. If that is the case my bet is the state actor being NSA/CIA maybe MI5 or Mossad.

[@sub-harmonik]

@@GreatTaiwan if you have a backdoor to every linux box running ssh? idk about that one

[@embedyt]

this xz stuff is honestly so interesting, crazy that some guy at microsoft only found it cause he happened to be benchmarking and noticed a 500ms difference in ssh login speed. if he never noticed we'd probably not know about this until it was way too late.

[@Papageno123]

He said he didn't notice it because the 500ms, he noticed it because of high CPU usage

[@orbatos]

Consider that it was noticed as soon as it was published and (responsible) server admins never ran this version.

[@sub-harmonik]

wow is postgres owned by microsoft too?

[@patrickoberholzer4278]

Sounds like you missed the point

[@FLasH3r]

@@sub-harmonik no, but it is used by a lot of major companies

[@TheMrChugger]

That maintainer needs the worlds biggest hug, support and love from everyone in our industry

[@aperson4051]

It is also a possibility that the maintainer created a fictional attacker so that if the exploit should ever get discovered....

[@asdfghyter]

@@aperson4051 in theory, it could be possible, but due to human nature it is _extremely_ unlikely. people don’t suddenly flip from being generous kind people into manipulative psychopaths like that, so they would’ve had to have built the project from the beginning for that purpose in that case. it’s way easier and cheaper to exploit an existing project with a maintainer that’s already tired than building something from the ground up just to tear it all down i guess a possibility would be that they were approached by someone and offered money for doing this, but even then it would probably be easier for the malicious actor to do the pretending themselves instead of relying on the acting skills of a random developer

[@DeadLikeYou]

@@asdfghyter Plus, theres plenty incentive to NOT do so. Why would someone like this willingly torch their own reputation and future employability to have some sort of state-sponsored attack after spending years (more than a decade?) contributing to open source, AND THEN go around and do as much as they can to undo this attack after discovery? Sure, its POSSIBLE, but highly unlikely compared to the higher likelihood of some poor dude getting socially exploited.

[@whannabi]

​@@DeadLikeYouWe don't know much though so all scenarios are plausible. Either way, any reasonable person would avoid working with that guy in the future wether he did it on purpose or not.

[@smnomad9276]

that maintainer needs a big check, he needs financial support. Stop being so cheap and actually help the guy, he doesn't need your "hug".

[@CFSworks]

I fully agree that it's unacceptable to be blaming Lasse or how the XZ Utils project has been run, and even from day one I was not seeing any significant deviation from the standard operating procedure. He was doing everything "the right way." But, human nature being what it is, most people are in denial of the fact that the FOSS ecosystem *itself* is what's vulnerable/targeted here, and they're desperate to fault XZ/Lasse for the attack to maintain that denial: "He screwed up by accepting weird PRs." (He did not, Jia was given full committer access.) "He screwed up by letting the code get overly complex enough for the backdoor's entry point to hide in plain sight." (It wasn't in plain sight, Jia added it manually to the release tarballs.) "The project shouldn't have been releasing curated tarballs, those should come from git-archive automatically." (Perhaps, but this was standard practice, not individual sloppiness.) Don't get me wrong, I think we're going to learn some valuable ways to change the "standard operating procedure" of FOSS to make it more resilient against this kind of thing even in the face of a burned-out maintainer and malicious co-maintainer, but we NEED to have these discussions in the context of the status quo not being good enough, rather than Lasse being not good enough to follow the status quo.

[@technokicksyourass]

I wouldn't stress about it too much. Security experts expect open source software to be riddled with security holes. That's why there are firewalls, intrusion detection systems and other such things.

[@ark_knight]

If Lasse is innocent, and I think he really is, his account will be reinstated. This is just standard procedures as investigation is done. But at the end of the investigation, I expect big companies to assign reliable maintainers to these core packages asap. And while at it, fund those projects. Because if bug bounty is a thing, then maintainers should be rewarded when niche packages are used.

[@jeffwells641]

@@ark_knight Will it? You work at Github in their security division? You have some insight we don't have? You must have more, please share your wisdom. If he's innocent, Lasse's account SHOULD be reinstated. You have no fucking idea if his account will or won't be reinstated. That's up to Github.

[@ark_knight]

@@jeffwells641 Yea, if Github (and MS) wants PR nightmare, sure. Unfortunately, they are not in vacuum themselves.

[@luckybutunlucky8937]

Linux and Mac users sweating.

[@ernstoud]

0:19 … “Some random Microsoft engineer”. Geez… that is really derogatory. He is a well known Postgres developer.

[@chocolate_squiggle]

I took the comment to just mean it was found by some guy not even doing work directly related to xz or sshd. He just randomly stumbled upon this from some other thing he was working on.

[@GeorgeSukFuk]

Stop clutching your pearls. It's Microsoft.

[@AsianNinjaGod]

"derogatory" lmao

[@satunnainenkatselija4478]

Many also call it a Linux backdoor even though it's in xz. Poor Linux getting a bad name for something he didn't do.

[@NiNgem-bb6lc]

@@satunnainenkatselija4478 I think that's because it was targeting Linux, there were conditions to only work there (I think)

[@abhishekgoel8251]

Yes, please make a fork if the maintainer is no longer maintaining the project instead of sending rude messages. Sending rude messages helps no one

[@overclucker]

I agree. Though there are shitty forkers out there too. libav and enough influence at debian to supplant ffmpeg, even though it was only a literal copy at that point.

[@alasdairclark4213]

The "good devs" know this, undoubtedly the attacker too - this wasn't a good guy though. The attacker didn't want to fork it, they wanted to control the original as it was a trusted fundamental pillar to SSH and if it was compromised would lead to [consequences]. If they forked it they would have to make it better and trusted, waiting years for community support and integration where they would have to keep the fork better than the original. The original already was all of those things and compromising the original was therefore significantly the quickest and easiest way to achieve this.

[@jonathanmacdonald9609]

As someone who would probably feel the same way if someone commented that on my project, that was barely rude at worst. The way we should see that type of message is not "Why aren't you keeping up with comments? Do you care that people are trying to use this?" it's "Yo, what's this? Looks useful. I need to know X though." (a week passes) "Hmm, nobody responded yet. Maybe it's dead?" (moves on with his day) In this case the attacker may have been intentionally been trying to make the author feel guilty. The later comments were clearly rude, and we should try to avoid accidentally hurting people's feelings when we can remember, but most of the time when people say something like this they're just wondering what's going on and they ask. It's not passive aggressive. There's no greater meaning behind it at all. Using this to justify bans is overkill by a lot.

[@grandsome1]

We don't celebrate maintainers of all sort of infrastructure in this world where everyone wants to be a creator. Mad respect for all maintainers around, it's a thankless job.

[@jamescollier3]

yeah. they attack all the men that built America

[@be1tube]

Every company that uses open source should contribute towards its maintenance - by paying or by employing people to contribute. Open source maintenance for widely used packages should be a well-paid gig.

[@nathansimons9881]

It's honestly infrastructure, like roads and bridges. Not even that, a lot of open source software is the underpinnings and girders of structures we all use every day. We don't get roads and bridges for free, we pay for those with taxes. The government should be paying software engineers to audit, maintain and write commodity open source software. There's a reason one of the major Linux conferences is called "Linux Plumbers" - it deals with all the "plumbing" that exists beneath the porcelain.

[@BlackMan614]

IBM did this at one time and I think Microsoft has started to do it?? Google also did this at one time but no longer AFAIK.

[@bobbybyrne1899]

If you work in a company, advocate for time and/or money be put towards the foss tools and libraries the company uses frequently. It's how the open source model is supposed to work. It's also a PR gold mine to show how your company is contributing back in meaningful ways. Helps attract talent as well.

[@TrimutiusToo]

Lasse is suspended because they suspected that Jia has access to Lasse's account I would think... Lasse is blameless here, account though might have been compromised

[@thighdude7]

Hopefully you are right - and this is a temporary condiition.

[@carnivorebear6582]

I would imagine it's because they suspended the xz repo and it's standard practice to suspend maintainers of such projects

[@mosubekore78]

What if Lasse has two personalities. He himself doing all of this? Unlikely.

[@ericm301]

@@carnivorebear6582 Yeah, probably just a knee-jerk reaction. If GitHub (MS) had actual people looking into this they could backtrack Lasse's access and check for anything suspicious. But, it's all bots now, and the pod bay doors won't open.

[@luckybutunlucky8937]

Linux and Mac users sweating.

[@Eagledelta3]

I do dislike that some are saying this is a risk unique to OSS. It's NOT. It throws into question the entire trust chain for all software devs. The way the attacker built trust for TWO years can be done in ANY organization

[@joseoncrack]

Yes and no. In a very small company, sure. For the same reason. For any large enough organization, it's very hard, unless there's a whole team of attackers inside, which is always possible but just a bit less likely.

[@cherubin7th]

@@joseoncrack No, I worked for a European government and your could literally go to a parking garage without cameras and use the exposed wires to pug into the unencrypted government network and read all the stuff. The government knew this and mandated full encryption but they were unable to get the project started for years. Also we had news that big corporations didn't fix several security bugs they had several times.

[@monishbiswas1966]

@@joseoncrack I suspect you could play the same tricks in a large company - the only difference would be the background checks for hiring may act as a bar, but with contractors and outsourcing this may be bypassed for anthing that is not high security.

[@anon_y_mousse]

@@joseoncrack It doesn't matter how hard a company vets new hires, as long as they're a malicious state sponsored actor, then they will introduce it and no large company is going to code review well enough to find everything.

[@not_ever]

@@joseoncrack I worked for a multinational engineering company where it would have been possible, in the not too distant past, to inject malicious firmware into all their products simply by having access to their factory because all of the machines on the factory floor had full admin access. They also had no source control and all the production code was on those machines. You would only need to be hired as a temp with no qualifications to f-up a very large part of their global business. Lots of large organisations have horrifically lax security policies, you have no idea what horror stories lurk out there.

[@frydegz]

the shitting on OSS from a lot of online (even security) influencer types is so weird to me. If OpenSSH were proprietary this would have taken many more months to find (if at all), would probably have been as slow as the backdoored version by default, would hide PAM behind an enterprise feature, and probably find some way to depend on javascript. The only reason this is even a story instead of an ongoing attack campaign is BECAUSE this was in OSS and had numerous checks before being put into a widely deployed LTS release.

[@timecubed]

No one is saying that these projects should be proprietary and closed-source, but people _are_ saying that the OSS community sucks and needs to get better, and that is a completely valid criticism of the OSS community. OSS is good, and it has its strengths, but that also doesn't mean it's completely impenetrable and doesn't have its flaws, and having people say that we shouldn't criticize OSS when things like this happen because "OSS makes software better" is just plain wrong, and removes the ability for anyone to improve the situation or the community. I do agree that sometimes the criticisms can get stupid, but this is completely valid.

[@ark_knight]

@ShibDivingSuit High entropy, obviously. Thermodynamics 201. People want drama and chaos.

[@descorchafairys]

@@timecubed this is absurd. They are making the dance for Microsoft that is trying to apply EEE. The same history. We, the people on this from the start, saw this kind of game over and over.

[@yurithebrave]

Yeah, came here to say this. I keep hearing that "this attack exploited an inherent weakness of Open Source Software" - what weakness is that? I'm sorry, I'm not a software engineer myself, but what is this weakness? In my mind there is nothing that would have made this type of attack impossible with closed source, maybe it would have been a bit more difficult to pull off, but it would also be dramatically more difficult to detect and remediate. I do not accept explanations that "it was the community that failed" - this type of software development should be governed by processes, so clearly there is either a gap in a process or human error was made. You can't point a root cause to collective responsibility. The whole narration around this feels like a weird psy op. I'm waiting for Apple to bring this up as the reason why people shouldn't be allowed to replace their batteries.

[@indiesigi7807]

@@yurithebrave Random unidentified people get in positions to commit code to important software. But but that could happen in proprietary case.. They would be formally identified and would be arrested. This isn't some conspiracy against open source, perhaps you're a bit too emotionally involved with the concept open source..

[@CodingGimmic]

Social Engineering hack was Kevin Mitnick's #1 skill when he was wanted and still alive.

[@mikescholz6429]

I didn’t even know he passed 😢

[@malvoliosf]

TIL Kevin Mitnick's dead. Pancreatic cancer, leaving a pregnant widow. Mitnick is not my favorite person, but that is sad.

[@blarghblargh]

now he is free. RIP

[@Naturalmedicineprescription]

and ?

[@greatcreate82]

I worked large crypto exchange with heavy security training....we specifically went over the story of a lone maintainer, too busy, who gets a life line buy a rock star, who after 6 months add's in his malicious code, to target companies down the chain. Be carfeful and vet your third party packages, and be weary of the lone solo maintainer

[@omg33ky]

I think lasses account being suspended makes sense temporarily. In the beginning it really wasn't clear if he was maliciously involved or not. We are quite sure by now he wasn't, and quite the opposite, but its better to just be safe. I really feel sorry for him and its sad it could even get to this but thats just the situation right now. I hope he gets his account unsuspended and can continue working on the project if he wants to and hopefully find some people to help maintain it.

[@krishnabharadwaj4715]

You are 100% right. I don't like people demanding stuff to fix things and get more features. If you want something, just fork it and do on your own. If you are not able to, just politely ask the maintainer. After this incident, every contributor will be thoroughly scrutinized for open source projects. Code quality over speed.

[@noobertime]

From a security perspective having a hard ban for certain types of behavior is a great idea. It limits the possible attack vectors for social engineering.

[@noobertime]

And to be clear those rude comments would have been an insta-ban on a project that I was managing. I don't have time for a-holes.

[@michawhite7613]

​@@noobertime Absolutely agree. If you can't make criticism constructively, then you're not making criticism at all.

[@dj_chateau]

​@@noobertime I think that could prove to be difficult since rudeness can be subjective to culture and fluency of a language. With OSS often crossing international lines all the time, it could be easy to misinterpret something as rudeness where none was intended. I still think we should make an effort in this area, but it may prove to be difficult to enforce and end up shutting out legitimately useful and productive contributors because they didn't understand the nuance of what they were communicating.

[@seeibe]

I used to be an absolute jerk and I still contributed some valuable code to the projects I volunteered for. Open source was one of the few ways for me to be part of something and find community, and I'm glad that wasn't taken away from me due to my mental health issues and my occasional bad behavior. Just throwing that out there.

[@michawhite7613]

@@seeibe Was there something that made you realize you needed to improve? I'm asking because that might be helpful in the future.

[@matytyma]

Nice update: Larhzu is no longer suspended

[@Fractal227]

This need more upvotes if correct.

[@matytyma]

@@Fractal227 You can see they even made a lot of commits to XZ and obviously removed Jia Tan from wherever it was possible

[@CarlosRangel]

awesome video, I feel terrible for Lasse Collin, he was totally duped by what could have been a state actor. Hope he understands he did nothing wrong

[@themodfather9382]

Or you're being duped by him?

[@The-Weekend-Warrior]

That "random Microsoft engineer" is a proper CHAD. Without him, this would have made it everywhere. That man is a LEGEND.

[@voidmind]

I hope security researchers can figure out heuristics to detect such attacks in other open source projects. And I don't just mean the source code, but the mailing lists and other channels of communication.

[@orbatos]

That can't really be done, sorry. This is a human problem and the attack was (likely state sponsored) espionage.

[@goraxe01]

We can't even get folks to not click risky links in spam. The industry is adversarial by nature ie as counter measures for attacks are deployed new attacks are developed. Consider how this social attack took place, there was some public discourse, however much of the trust building probably occurred through private channels. For any security measure that would mean having to monitor those private conversations. I have some thoughts on addressing at risk dependencies in the supply chain... However I suspect much would be manual review

[@orbatos]

@@goraxe01 exactly, and this is something that open source makes possible, though still by no means easy. Just like the xkcd we have corporations worth billions running on technology they often make zero effort to help maintain. Some pressure in the name of security is appropriate. If this was an internal product we'd be hearing of what losses were incurred, not how it was found and prevented from causing issues.

[@SimonBuchanNz]

To some extent, the actual blame here is the distribution maintainers that aren't verifying the developer, and doing the reviews that you would do for a new maintainer on seeing a hand-off.

[@orbatos]

@@SimonBuchanNz No, not really. This was espionage by a contributor who was under cover for two years. There's no way for package maintainers to verify in cases like this. What would have helped? The dev getting support from downstream projects rather than being forced to run it by himself.

[@krellin]

in one of my former companies i had a CTO who set up build for every single dependency we had, everything that was open source our system would build all the artifacts and package them with our stuff... now i see why he was doing it. Its only true way to be sure your weakness is actuall literal malicious code in the open source code and no one has noticed it... you wont be subject of an attack when someone modifies the artifact only...

[@goraxe01]

Clean build rooms are going out of fashion. If you want some nightmare fuel look up Ken Thompson compiler hack

[@GreatTaiwan]

could u explain more

[@krellin]

@@GreatTaiwan basically we had pure java/kotlin stack, and mostly used open source projects, he was CI/CD nerd, he built a system that literally checks out the projects of jars we depended on, builds them from source, stores the artifacts in our repo and compares checksums with maven repos... only then our services can use that to build themselves... and any time his build process would break he would carefully observe wtf changed that suddenly it does not work. It had 0 impact on actuall speed of our builds cos once he built those jars those are cached in repo... but we knew we built them so no man in the middle swapping arts would do anything to us. XZ was exactly that... hacker does not show obvious signes of the hack in the code but outside (in build process).

[@complexity5545]

​@@GreatTaiwanI guess he means compile, checksum, and fuzz test everything yourself. Compile all ports for yourself and virus scan the compilations. I spent sunday and monday checking compilations. I have about 9 computers and 2 of them were dedicated to kali. I just finished testing [ haveged ]. I don't trust anything right now. I'm one of the few that has 1 epyc-64 256GB-RAM and 1 epyc 1TiB-RAM. I'm trying to reduce the kernel footprint. I'm shifting stuff to freebsd for a while. I'm scraping my mailing list. This event is going to have me backtracking for about a month, just setting up compilation tools and chains to check for this new move. But its a great find for the community. But now the new black hatters and script kiddies are going to try this "long game" now. In my effort to fight this cve, I was surprised to find other areas and affected projects. Give the maintainer of xz a prayer. He might be getting blasted with messages from all types of security companies. This plus the rumors in the security community about russia causing concerns in the tech world, its all looks like a huge coerced play from a certain war. I guess ww3 begins in the tech. Ha...I'm laughing but I'm kind of serious.

[@sevenismy]

It would probably not help here since you would probably use the same compromised xz build files/system in your CI/CD

[@crimiusXIII]

I literally printed and pinned that XKCD to the wall in our cubes a month ago. It's so real.

[@ktkrelaxedscience]

"Be kind if you want to be talked to." That's a sentence by which to live life.

[@vogonp4287]

Given the effort involved, I am almost positive that this was state sponsored. There was so much effort and time spent to build up trust and the potential of many sock puppets. I can't think of many reasons why someone would be so dedicated to make an attack like this otherwise.

[@passantNL]

People do all sorts of stuff just for giggles. Take Linus Torvalds. He had no reason to think that anyone else would be interested in a kernel, yet he spent his spare time in university to write Linux. I'm pretty sure he never expected it to be of any use to anyone.

[@vogonp4287]

@@passantNL True, but this is a very specific attack vector, and seems like it would have cost a lot in terms of time and money to actually get to this point.

[@v0ldy54]

Well, a backdoor like that is incredibly valuable, the money you can make off it alone seems enough of a good reason to justify the effort.

[@tinkletink1403]

@@v0ldy54 follow the money, as they say

[@davidprock904]

For all the unrecognized, trivial, and tedious work that you do that goes unrecognized, I personally truly do thank you for it!

[@Imperial_Squid]

God fuck that "hey is this still maintained?" thing hit hard, I'm by no means an accomplished open source dev but at one point in uni i thought it would be cool to add TTS functionality to zotero the reference manager. Uni being what it is and massive burnout meant I ended up putting the project on the back burner after only a month, it's barely a skeleton and isn't fit for release. But since the repo is public people still find it and ask me about it, I've had two emails and an issue just in the last couple of weeks asking for updates and I feel so fucking bad because I'm too busy job hunting at the moment to really do anything on it, but I still feel like i owe it to these people, i can't imagine the mental stress being a maintainer on an actual big project would bring, fucking massive digital hugs to that guy.

[@EsperSpirit]

If it's barely a skeleton and unfinished put a note about it in the readme so people stop pestering you imo. The way I solved this was to setup a proper CI-pipeline with Github Actions, so it's easy for others to write PRs with tests. If someone asks for a feature or fix, I'll tell them to provide a PR and I'll merge+release it. So far people haven't provided anything because it wasn't important enough for them after all. If it's not important for them, it definitely isn't important enough for me.

[@Imperial_Squid]

@@EsperSpirit it already has one, put there 8 months ago

[@SomeMorganSomewhere]

@@Imperial_Squid yup, most people don't even bother to read the README before they come whinging...

[@freeideas]

If this can get into code that everyone can look at, imagine how many massive back doors must be in closed-source code. The number of people you have to trick or blackmail to get your back door into closed-source code is vastly smaller.

[@trueriver1950]

I'm not sure you are correct. I'm a huge fan of open source, but here the social exploit depended on open access to just one maintainer. How can you justify the claim that other development environments have a smaller number? In a closed source environment the number of people with this kind of access is smaller than in FOSS. That brings huge benefits, but we must wise up to the fact that it brings along with it the increased risk as well So the attack surface for social engineering is larger for an open source community than for a corporate environment with corporate safeguards. However: the other side of the coin is the RMS thing about not having blobs in the code. That is an essential safeguard for open source that somewhat mitigates the fact that the doors are wider open for attack. All the more important to resist closes blobs where we do not have source access. And that is the strength of a fully FOSS system. So both attack and counter attack seem to have larger scope in open source: and this exploit demonstrates both. In my analysis it is a judgment call as to which dominates: the security benefits of more eyes looming as against the disadvantage of debts being more accessible and less supported than they would be in a closed corporate environment. Don't let the meme about "open is good because it's easier to spot attacks" blind you to the things we need to learn about the corresponding increased vulnerability to social engineering. And that's why this particular aspect of this CVE is being focused on in this video: and kudos for this channel creator for pointing this out.

[@dotanuki3371]

@@trueriver1950 not sure if I buy that OS automatically is more vulnerable to SE the US gov can march into any of the US-based tech companies, say "put this code into yours", and "if you talk about this, there will be severe legal consequences". that's not even SE, that's just leverage. as for other govs, how many employees with family in india or china do you think works at any of those companies? probably not few, right?

[@freeideas]

​@@trueriver1950 We will never know whether I am right or wrong about closed-source, because it is, um, closed. ;) Also, I am not saying closed-source has fewer maintainers; I'm saying that closed-source has fewer people who can see the code.

[@freeideas]

@@dotanuki3371 You bring up and even stronger point than the one I made: closed-source is defenseless against government manipulation. If government tried the same thing with closed source (and, btw, this is likely what happened in this case), someone in a different country can see the exploit and remove it (also, btw, probably what happened in this case). No doubt EVERY major closed-source implementation has government-manipulated code.

[@StarkRG]

@@freeideas Closed source may have fewer people who _can_ see the code, but, in general, they have more people actively looking because that's what they're paid to do.

[@freeideas]

No doubt the closed-source world has back doors just like this except even bigger and better. The difference is, you will never find them.

[@Vin50000]

people like you who treat open source and closed source like a crusade are so embarrassing. You dont care about security you care about your "team". If you're gonna be like this at least do it for something trivial, instead acting like a bot posting essentially the same comment multiple times

[@travisporco]

I'm sure you are right.

[@polettix]

@@Vin50000there's a lot of places where this whole XZ thing is portraited as a "problem with open source", so the distinction between "teams" is very tangible in the whole conversation. To a big extent it is, because "hobby and unpaid" ne "work and paid". It's worth remembering that this can happen in the closed source world too, where finding out the reasons why there are delays and weird CPU usage is practically impossible.

[@caerphoto]

@@polettix but it IS a problem with open source. Bringing up closed source is just whataboutism, it's just a distraction from the actual issue. Nobody worth paying attention to is suggesting "this wouldn't have happened with closed source", they're just saying that the issues brought to light by this hack need to be addressed.

[@aerahtv0000]

​​​​​@@caerphotoas OP said, I am pretty sure there are even bigger exploits in closed course software. Do you think that some hacker couldnt get hired into some company and do the same stuff? I think it is even much easier to do, some companies completely trust to their employees and try to release features as soon as possible without thorough testing. closed source does not have so many eyes, sometimes it needs only for 1 guy (team lead or whoever checks the code) to miss something during the code review

[@mitchellmnr]

Theo, I think your video has actually been one of the best so far. You got the best of details with LLL, went over the higher details as well as backed the original author. Well done, really felt pissed off with you. As someone on the OSS world, I 100% understand and can relate to him. I'd love to help the creator in ways where I can like you have offered - but at the same time we all need to be careful. The reason I say that is ... well, I could say who I am and show my creds and all the infra I have avail ... but ... can you trust me? - dumm dumm dummmmm..... so a monetary assist would probs be the safest. Let us know if we can help once you hear back from him :)

[@Xankill3r]

I see the underlying social engineering exploit as a systems problem. The free lunch is over. If corporations like Microsoft, Google, Apple, etc can profit massively from open source they can start pooling resources to help maintain these projects. Either put more funding into the Linux Foundation and have it steward these projects or create a completely new foundation. PS: Recent example of free lunch. Apple's game porting kit is based on a ton of different FOSS projects - none of which are actually receiving any help from Apple at all.

[@middle_pickup]

A rando SSH benchmark. That's all that stood between this exploit and the world. Yikes.

[@trueriver1950]

Massive approval for this use of other channels' content here: well chosen and (importantly) well acknowledged too. You've avoided the trap of trying to do everything yourself.

[@andygaal7936]

I really like your take on the issue, very informative and inspiring video

[@Patashu]

The payload being snuck in with test data is so sneaky. I definitely wouldn't check that during review if it were my project. And apparently by the time the payload was grabbed by BASH scripts he was committing directly to the repo, no review, so it was already too late to do anything about it.

[@JorgWMittag]

As I understand it, it's even more sneaky than that: the code which runs during the build process and injects the backdoor into the compiled binary isn't even in the Git repo. The code uses GNU Autoconf and GNU Automate to build, but - and this is pretty common for projects that use the Autotools - the repo does not contain the ./configure script. The repo only contains the configure.in sources, and the ./configure script is auto-generated with Autoconf when the release tarball is built. Except in this case, the ./configure script in the release tarball contains a little bit extra code. This is a fairly typical setup. You don't want to have generated files in the repo, so you keep ./configure out of it. But, Autotools can be finicky to set up and have a lot of dependencies, so you don't want to impose this on your users either. So, you build ./configure using Autotools and include it in the release tarball. Many projects do this. Now, how often do you check that the ./configure script in the release tarball is byte-identical to what you would get if you ran Autotools on the source code in the repository? I'll put my hand up and say that I have *never* done that.

[@Patashu]

@@JorgWMittag Oh yikes. Maybe we should swap to something that we ARE comfortable with 'imposing on our users'... Seems like a serious failure of the tools and programming language if no one actually wants to read or build it. (I say we as if I have any stake in the matter; I already fled as far away from C/C++ as possible!)

[@vickmackey24]

Are we supposed to be believe that this was the one and ONLY time that anyone has ever surreptitiously added a backdoor or malicious code to the linux ecosystem this way? I'd be shocked if this was the one and only time, and that we just got lucky that someone caught it almost immediately and made it known. And it doesn't even have to involve hackers. It could just be a "trusted" source/maintainer who always had malicious intent, or eventually became malicious, or was blackmailed to become malicious. This should be a real eye-opener for people, and it's probably even worse with closed source. The community should figure out a way to audit and inspect everything in a way that makes this kind of thing much more difficult. We should leverage our new AI friends for the task somehow.

[@nibnob9]

shocked!

[@garymuller9771]

you did a very important job here to shine some light on this side of the story. Tank you for that. And i wish the original maintainer to get through this all right.

[@Jakob.Hamburg]

Nice video, well done. I come from a very different field and this is a nice insight into the happenings in the OC programming community.

[@benmuschol1445]

this was a great video but you dont really get to frame it as "what everyone missed about the hack" when you are mostly reading someone else's article lol lots of people were talking about the exploit but tbh the open source social engineering side has been the most discussed part!

[@danieltr94]

Imagine a non-profit that financially supports developers of popular, widely-used projects, at least in the U.S. There could be some kind of yearly audit to prevent people from milking it, but at least some kind of consistent 'income' is provided for maintaining these essential pieces of our infrastructure.

[@seeibe]

I'd prefer a world where we don't need to struggle for survival and actually have the time and energy to do this sort of thing as a hobby. But I'd settle for the non-profit in the meantime.

[@lucyinchat]

OpenSSL is maintained by OpenBSD

[@MrVecheater]

The German govt does that recently for a few projects

[@7200D2KJA]

Funny: this is how most EU countries deal with social security. Tax paid by citizens is applied to funding healthcare or other initiatives which serve the public interest. In the ultra liberal US however, this sort of thing is considered ‘evil’ or ‘communist’. The results are plain: social inequality, physical and mental health issues and indeed situations like this one…

[@autohmae]

The Linux Foundation already does this, BUT they hadn't noticed how important this project had become.

[@dbjlmc]

Theo this was so well done, thank you. Subscribing now.

[@user-ru5bd7vn2w]

Thanks for discovering this social part, that's interesting for me then the technical part that I cant fully understand

[@Oktokolo]

What this really shows is that we really need to invent a way to eventually finish software. The exploit is now rpoven. State actors with unlimited budgets will do them all the time now just to be sure that they have a backdoor when they need it. The constant need for maintenance on a constantly growing amount of code makes defense against this sort of exploit near impossible. The only software that is immune is software that's fully done and has been proven to be correct by multiple parties as it doesn't need maintenance and therefore it needs no maintainer that can burn out. Also, distros should finally stop using tarballs. Build from the official release commit. GIT already is a tamper-resistant write-only message chain. If you know the hash/ID of a commit, you know the full state of everything cvovered by that commit and all commits in the chain back tio the beginning.

[@BrokenKanuck]

Random Microsoft engineer? Are you kidding? You're talking about the primary developer for Postgresql.

[@user-pc4sb1dg5q]

Then spends 20 mins talking about being respectful

[@Galakyllz]

Awesome video. Thanks for highlighting this aspect of what happened.

[@agvulpine]

Thanks for this video. As you explored and discussed the stress and wellness of the maintainer of xz, I was reminded about the story of pkzip and how that ended in 2000. rip Phil Katz

[@ivolol]

As long as the OSS ecosystem stays in relatively the same form, "we need to do better" is going to ring as hollow "You've gotta do better, senator!" from whichever Marvel TV series that was.

[@andrewdunbar828]

"Chaos" isn't the word I'd use to describe this. That implies everybody running around screaming and yelling and not knowing how to proceed.

[@andrewdunbar828]

Actually, watching the rest of the video hours later, I see that you are right! I didn't know about that chaos until now.

[@thekennethofoz3594]

One of the best KZbin posts I've seen in ages. Thanks for showing both compassion and fairness.

[@ovflowd]

Thanks for covering this, Theo!

[@chrisalexthomas]

The mean and nasty messages on the mailing list are mean and nasty on purpose. They're exploiting his fragility to push him to make a mistake and throw his hands up and welcome the attacker into his arms and trust him. Thats different from other people who are just mean, nasty, and feeling privileged on typical github issues I've read. This was a deliberate attack on his mental health. They knew if they pushed him hard enough. He'll break. Their plan will be greatly helped by this.

[@marinrealestatephotography]

I just want to say THANK YOU to anyone and everyone who helps make Linux a viable operating system!!!

[@metatechnologist]

Excellent video. Everyone needs to see if.

[@CoriolanBataille]

That story is insane … big up to that maintainer 👏💪 stay strong

[@entelin]

As for github suspending the original authors account, that's the right initial reaction to have. It's nothing against the author, the first thing you do in response to these things is cast your net as wide as feasible and shut it all down. *then* you investigate and open things back up in a way you are confident about.

[@truehighs7845]

We don;t even know what the maintainer means by "long term mental health" because if you run a package that so seminal to Linux for so many years without even making a penny and eating just shit instead, I would have long term mental health issue too, is there more social engineering in the background, like sapping his financial foundation with other attack vectors.?

[@user-pl1pq5nz2b]

very quick to cover this! nice!

[@johnxina1681]

lol

[@RonNorthcutt]

The more I learn about this exploit, the more convinced I am that this had to have been state backed. While this is terrifying, the really scary thing is that this is likely going on across dozens if not hundreds of projects, and there may be one or more existing backdoors active in other systems right now.

[@Fighter178]

We need to give that poor maintainer a giant hug

[@nibnob9]

is that what you do when someone collapses your society?

[@rweninger]

Jia Tan also forked lz4 and zstd. Wonder if those repos include exploits too.

[@Cuptial-ev9tb]

We like having Low Level Learning in your security vids. Hell yeah

[@Raftzard]

Thank you for the excellent analysis! Subscribers=Subscribers++

[@pwall]

I kinda knew that the video was gonna be about the social engineering part, but I do think some techincal people (like @BrodieRobertsom ), did address a bit this part!

[@pedro4205]

I think what this video is about is more on the human side of the execution, not just "it had social engineering involved", but about the harass and toxic relations.

[@pwall]

@@pedro4205 Yes! I am not invalidating this video, I actually quite liked it. Just was answering the "I don't see enough attention" which I just came out to say another creator that did address (to a lesser extent) the topic of this video.

[@PubRunner]

Imagine how bad the Maintainer feels atm …. Has anyone reached out to them to make sure they are ok? This pissed me off so much.

[@vegn_brit5176]

Thanks for this insightful video.

We *really* need to make a list of all the projects that are like xz. Cornerstones and have only one or a few maintainers. That way we can direct more resources and/or keep an eye out for similar tricks.

[@hodgenick]

State level actors

[@as-ng5ln]

I feel bad for the original XZ maintainer.

[@nicholaushilliard6811]

Thank you for giving open source dev version of what really happens

[@cherubin7th]

Crazy how such a long planned attack got caught before it even made it to stable.

[@boredbytrash]

And crazier? Just because some ssh logins took a bit longer and were more CPU bound… pure coincidence and luck that this was found. Also: that the engineer who found this directly went to the OSS community and shared his concerns instead of keeping it to himself or even just ignoring it…

[@JorgWMittag]

There was a bug filed against systemd recently which raised concerns about the large number of dependencies it links against. Developers are actively working on fixing that. This would make the exploit useless. It is thought that this work accelerated the attacker's timeline, prompting them to make mistakes. Remember, there is a pretty brittle chain of dependencies this attack relies on. The exploit code is in liblzma. systemd links against liblzma. OpenSSH upstream actually does *not* link against systemd, precisely because the developers want to keep the dependencies small(ish). The patch which links systemd to OpenSSH was developed by Linux distributors because of some bug reports that were filed with OpenSSH sometimes failing to be properly restarted by SystemD. For OpenSSH to send status messages to systemd, the easiest way is to link against systemd and use their helper functions. If any of these domino bricks gets removed, the exploit doesn't work. If OpenSSH does no longer link against systemd because the helper functions get broken out into a separate library, the chain is broken. If systemd no longer links against liblzma because that part gets broken out, the chain is broken. If the OpenSSH patch is removed because either upstream OpenSSH or SystemD gets more clever about service management, the chain is broken. Essentially, if they didn't get the exploit code into this LTS release of Ubuntu, they wouldn't get another chance.

[@mafriese5]

Hello, security person here 👋🏼 The thing I‘m a bit worried about now is that A) we don’t know what other projects have similar backdoors B) this incident might lead to longer patch times. Companies take their time to patch their systems anyway but this might make them even more suspicious of new versions and patches. C) even big corporate Linux maintainers like red hat are not safe from this (because fedora was affected). This is probably the most worrying thing about this. I cannot imagine what the damage would be if this would’ve been part of RHEL

[@JernD]

Fantastic video, thanks so much for this

[@dannyisrael]

Nice. Absolutely appreciate seeing this.

[@surfingbilly9654]

The problem isn't really with the maintainer at all, the issue i guess is the system as a whole, still is absolutely insane to me that someone who makes something that is the backbone of modern computing literally receives 0 compensation for it and is expected to maintain it in perpetuity for zero compensation as well. How could you not expect something like this to happen in that case?

[@tecoberg]

You're right. The original maintainer is a victim as all the open source community.

[@AbDullAHMoHAAmeD]

Thank you for explaining this unknown beautiful community to me thanks for all the Devops working on opensources ❤❤❤

[@bmjhayward]

Great vid great points thx Theo. Maintainers are saints of the digital era

[@msclrhd]

Thanks for making this video, and thanks to everyone -- including Lasse -- for maintaining and writing the software we rely on every day. Maintaining software is a hard and often thankless task (I know, I've done it). Maintainers are legends. While this event shows the lows of open source, it shows the highs of open source, how everyone is pooling together toinvestigate this and potentially other issues.

[@samiraperi467]

Remember the Ken Thompson compiler hack?

[@christopherkirkland7174]

Well done. It is important to realise the importance of people and particularly their health.

[@timothyvandyke9511]

I appreciate you for appreciating and not blaming the good maintainer. Lasse Collin is a hero

[@karenwest6350]

Man, I feel so bad for Lasse. I can't even begin to imagine how awful it must feel to be stuck with a vitally important project with no support, getting burnt out while unhelpful people shout at you, finally getting some help for years and years and building up trust and being thankful, only to suddenly find out the person you trusted nearly got away with an attack that could have utterly destroyed companies. Imagine how bad getting catfished feels, and imagine how bad it must feel when you getting catfish was nearly worth billions and all eyes in the world are now on *you*.

[@Z3rgatul]

People who say "open source is the best, super secure, super reliable" probably never committed anything to open source, not even saying about maintaining a project. Their mindset is like "there are some good guys working for free for me, so I can use their work and tell everyone how good open source is" Arguing with them doesn't usually lead to anything

[@TetrisMaster512]

I've been using Linux (and BSDs) exclusively for almost a decade and a half, have spent about as long in communities focused around FOSS, and have contributed a bit to some projects. The claim (which I would personally make) that open source is the best is usually that it's the best model (at least for the end user), not that the software itself is necessarily the best option in terms of functionality. Claims of superior security usually just revolve around the fact that code is open to audit, this incentivizes sound design in the first place since security through obscurity (i.e. making a poor design and hoping nobody ever actually figures out how it works) is even less viable in FOSS. I know this still requires people to actually perform the audits, and FOSS is still just as susceptible to security issues introduced through programmer error. Personally, I'd focus more on the privacy aspect of security, which is a pretty clear win for FOSS, given how it's much more difficult to implement forced telemetry in a FOSS project. Agree that there are entitled users who don't contribute back. I think some of it is an empathy issue, kinda like how working at a retail job, some of the worst customers to deal with are those who have clearly never worked retail.

[@BlueEyedVibeChecker]

MacOS: Less exploits than Windows and Linux combined, fully closed source. Linux only has less than Windows because no one wants to waste time hacking

[@lucass8119]

@@BlueEyedVibeChecker I hear this "nobody would waste time" argument all the time, and its bunk. You have way more linux devices in your home than anything else. That smart fridge thats a vector into your home network? What about your router? And then servers too - which, you know, actually hold data worth hacking. I mean its not even close - for every piece of critical infrastructure running Windows there 10 running a linux operating system.

[@MikkoRantalainen]

I don't consider this as a failure of open source. It just another example of XKCD 2347. Everybody using a piece of software some lone person maintains as a hobby, maybe. This happens with closed source software, too, but the problem is hidden beneath the surface there.

[@samhiatt]

New subscriber here. Thank you for standing up for the humans who create and maintain open source software what we all benefit from.

[@HoggyMayhem]

Don't do it out of your own pocket, Theo. Let's do a fundraiser for Lasse.

[@stage6fan475]

I think this maintainer was up against the full weight of a state sponsored organization that has huge resources looking for flaws, both social and technical. Probably China. Excellent video on this.

[@alexyo2440]

It's actually Israel but close enough

[@funtechu]

Someone give Lasse Collin a hug.

[@brentsaner]

There is a reason why it's extremely difficult to get backdoors in the Linux kernel, and even harder into the OpenBSD kernel. Maintainers do not owe you niceness, acceptance of your contribution(s)/effort, explanations, nor patience. Let this incident show why.

[@henryhansen3662]

Binary blobs are a security risk.

[@knghtbrd]

This was absolutely planned from the start, who knows how many more times this is ALREADY under way. Thing is, the bad actor was offering to help, and it was not going anywhere. Then socks show up and start berating the dev FORCING HIM TO BURN OUT and hand over control in part and then in full to the bad actor. And if you don't think the bad actor was a government, you're not paying attention to the YEARS LONG exploit. Ain't no skid pulling that off. The question is … which government? Indonesian IP. But that doesn't mean anything. People are thinking China. Some are thinking NSA/CIA. I'm thinking YES.

[@timothyrawlins6382]

Making attempts at this are certainly cheap, you can offer to help with a lot of projects given a few disposable identities, if you get taken up on too many "burning out" or unexpected family/ work commitments are easy outs. But actually doing this is fairly expensive ( you can't have that many people spending 2 or 3 years building trust doing maintenance drudgery especially with the limitations on needed skill sets ).

[@Rebel101]

Good post. Thank you!

[@fu5ha_edits]

Oooh. I guess because of the specific crowd i follow on twitter, the people perspective was the thing i saw the most frequently. But, the point about banning rude people as a security vulnerability is such an interesting one that i hadn't seen yet! Thanks chatter!

[@JSiuDev]

This is the worst nightmare for a opensource maintainer.

[@eduardoandrescastilloperer4810]

The attacker did something that looks straight out of a horror movie or like a psychopath character

[@FadkinsDiet]

Ehh, it was just a slightly more educated version of good cop bad cop

[@eduardoandrescastilloperer4810]

@@FadkinsDiet From a certain point of view yea, but definitely smells like some sort of psycho manipulation