Jonnathan Blow Predicts xz Years Ago..

[@edumorangobolcombr]

If aliens invade the earth there's probably a JB clip complaining how intergalatic security is such a failure

[@jeezusjr]

JB gets out of jury duty by simply making sounds come out of his mouth hole.

[@Handola]

justin bieber?

[@pastenml]

I think that's more Terry A Davis domain.

[@SmeggmaMan]

he'd be right too

[@Nyxar-2077]

​@@Handolahahahaha

[@tinrab]

I mean, if you constantly complain about stuff, you're bound to be right about some of it.

[@g3nko0]

Jonnathan Blow about problem with Jonnathan Blow

[@tauiin]

yeah lol, A lot of JB stuff i've seen is just "I Dont like ${thing}"

[@AximVidya]

boom, roasted 🤣

[@meanmole3212]

Yes because if I am observing worrying behavior from Russia for many years and predict in 2020 that a war is imminent, that's just me randomly complaining about stuff and being lucky with my guess.

[@raymondhill7837]

lmao this

[@top0657]

To be fair it didn't take 2 years to notice the backdoor. Jia Tan just spent 2 years building up trust with the maintainer while at the same time studied carefully what kind of changes received far laxer reviews from Lasse. The backdoor actually got noticed in just few weeks after the release. Yes arguably by a lucky chance but imagine similar situation happened with a closed source software. Even though someone could observe the same symptoms that led to the discovery here, no one could have investigated the issue. In that scenario the person would propably have just reported the issue to the developers who then could have fixed the symptoms making the backdoor even more difficult to get noticed

[@dansanger5340]

Having access to the source makes it easier, but people investigate security issues without source code all the time.

[@enkiimuto1041]

And it got noticed before it hit mainstream too.

[@jon9103]

​@@dansanger5340indeed, but that reduces the chance of success. Obscurity works in the favor of bad actors.

[@caleb-af]

@@enkiimuto1041 I believe this is due to sheer dumb luck. If the exploit didn't significantly affect the execution time it would have gone under the radar.

[@joshallen128]

we still dont know if both of them were in cahoots, or if jia and lasse were one and the same

[@demmidemmi]

Using this attack against open source is so disingenuous, we have seen close sourced companies fall victims to much less sophisticated attacks with much less planning and being owned for way longer. Has Prime forgotten the SolarWinds incident?

[@khaoscero]

preach

[@ForeverZer0]

This. OSS does not have the "perfect" solution, and many of its criticisms are valid to varying degrees, but it is still miles beyond anything proprietary and/or closed source. It is also disingenuous to always frame OSS as software that anyone from anywhere can just push code into. I have rejected plenty of PRs to my own projects simply because I felt they were outside the scope of what I wanted it to be. Not every projects is out to accept any 'ol PR that adds a new feature, I am not alone in wanting to keep the scope of my projects well-defined with a reasonably minimal API surface, and I know I am not an exceptional case in this regard. Not every project is trying to be a Swiss army knife.

[@bk1507]

Tru. The problem and the solution to xz ended up being open source

[@daphenomenalz4100]

​@@ForeverZer0 yes, they took advantage of a dev who was tired. This was a mental attack too not just an exploit.

[@copium5050]

you mad bro???

[@ttrev007]

I think open source is still safer then private. I don't trust private entities to not put in backdoors and with no way to check they can always get away with it. At least with open source their is a chance that someone can find the backdoors when you find something suspicious. More independent eyes look at the code when they want to make contribute to the project. if you want to use a project and need good security you can learn the code to verify that it is safe, not so if the code is private. Open Source is not perfect but i still think it is superior to private code system.

[@jsonkody]

exactly, proprietary is much more smelly

[@markmywords3817]

​@@SimonWoodburyForgetnot to mention that I big software companies are probably being infiltrated by state actors via the same social engineering and we may just never see them being reported on.

[@scottwarner7729]

It's genuinely not.

[@Kane0123]

Rubber hose attacks are effective against both.

[@tedchirvasiu]

​@@SimonWoodburyForget The reverse also applies. It's because those people aren't paid to review and maintain code that they will LGTM it more easily or take months to merge PRs. You still imagine that open source is this thing where every project is maintained by tens, hundreds or thousands of highly enthusiastic people and whenever some rando spots a bug he goes to submit a PR instead of complaining. But we have just seen with xz, core-js and others that many such projects are maintained by a single burnt out person who is doing a thankless job.

[@spooky_action]

We should probably purge our repositories of random encrypted binary blobs

[@totojejedinecnynick]

and while we are at it, could distros please compile source code before they ship it? Just fetching tarballs that do not even match presented source code is clearly is not the best idea...

[@wanking9040]

and flag all usage of 'eval'

[@Kane0123]

@@totojejedinecnynickI was actually thinking about that - why would they not be compiling from source?

[@potato9832]

@@wanking9040 You could just hand roll "eval" with system(). It'd still accomplish very similar functionality. And you could do funky things like system("gdb --pid=123") and attach to yourself.

[@shadamethyst1258]

@@Kane0123 Because it takes forever, especially on lower-end hardware. What you want is rather a way to verify that the compilation was done correctly, not to do the heavy lifting of actually compiling it, but I haven't heard of anything like that yet :/

[@snooks5607]

12:41 Andres Freund was also paid by Microsoft to benchmark these opensource projects, true it likely wasn't directly his responsibility to go deep diving into this particular issue but point being that many big companies now rely on open source projects so they are actually paying people to keep an eye on them similar to their in-house projects

[@AuthenticGadzooks]

That is very true, but sadly the people helping develop open-source projects rarely get sponsored by companies to help maintain their projects.

[@sporefergieboy10]

denial anger bargaining

[@christiangonzalez6945]

​@@sporefergieboy10 Loled

[@daphenomenalz4100]

Not only this was an exploit, this was also a manipulation of a tired dev ☠️

[@segueoyuri]

I don't get what all this fuss is about. The dude worked for years to get this inch by inch and then only beta versions of linux with the most updated version of xz were affected, most of linux distros stood just fine because they weren't updated to that point yet. If you use a stable linux version of literally any distro, you're fine. Sure you can get a lot of data out of 1 hour of active backdoor but really? Prime is acting like linux & open source is unusable and done forever lol the community did exactly what you'd expect it too, someone caught it in a couple of days and everyone affected rolled back to a safe version.

[@OpenGL4ever]

@@segueoyuri No! Finding the bug required two requirements, the first was skill, the other was pure luck. The evil code was already included in Debian Testing, for example, and no one noticed it yet. If the German Microsoft developer (-> skill) hadn't worked on it on his flight back (-> luck) to the USA, it wouldn't have been noticed and the code would have ended up in Debian stable later. There was no one who discovered the evil code when committing to Debian Testing.

[@segueoyuri]

@@OpenGL4ever I don't see any luck in it. Linux/open source has an army of the most capable people in the industry looking at and messing with the code. All day, everyday. Every bit of it is deeply scrutinized. IMHO it was inevitable someone caught it. That's not luck, that's open source by design. Sure there's skill, but I guess it's fair to take the skill part for granted when you have literally all the supercomputer OS teams with their hands on Linux, for instance. And to my point, you confirmed it was caught in Debian Testing, so not a stable version. And even a Microsoft guy, who's not even involved with open source. We also seem to be getting the story from different places. From what I heard it was a guy benchmarking his server and noticed sth was giving him a CPU overhead until he pin pointed the origin to the malicious code. [we kinda do have the same story though www.theguardian.com/commentisfree/2024/apr/06/xz-utils-linux-malware-open-source-software-cyber-attack-andres-freund] But yeah, who knows what the team behind it was able to get in the period while it was active. What I'm saying is, comparing to the effort of slowly inching for years towards it just to see it crumble in a short while and just affect testing and beta versions... It's not like I'm all that worried tbh

[@jwadaow]

@@segueoyuri It wasn't long ago an issue was discovered in sudo...

[@segueoyuri]

@@jwadaow sudo is an issue in and by itself. What are you talking about?? lol "Oh my system was just fine it just had this one teeny tiny little breach where somebody outside was able to get sudo" ?????? AHAHAHAHAHAHAHAHA

[@robfielding8566]

I came into work one day, and Jason Wright who was the guy that wrote our ethernet drivers, was accused of backdooring the FreeBSD random number generator for the US government. Theo De Radt stopped everything, and his commits were evaluated. Jason had to make a public statement about conspiracy theories. I wish I could get accused of such a dastardly act.

[@ThePrimeTimeagen]

What an honor.

[@TheHorse_yes]

Also, that's Theo De Raadt for ya. His usual 5/5 attitude at work.

[@lifelover69]

@@TheHorse_yesTheo is OpenBSD, not freeBSD. But funnily enough, Theo's attitude did lead to DARPA funding being canceled for a security project named POSSE.

[@autohmae]

Blow has a pretty good idea, but their is a huge missing piece in his reasoning: this was a multi-year social engineering job and the state-sponsored groups do the same in commercial companies.

[@Kane0123]

He literally talks about it and recognises that it’s a problem in closed source but that accountability is more explicit and impactful at the individual level.

[@herlegz6969]

And every day trillion dollar companies leak personal information and nothing happens to them. The system rewards nefarious and negligent and punishes the innocent caught in the ponzi scheme.

[@marcossantos1998]

He's whole argument wasn't that it does not happen in closed source, but rather is that it's way easier and cheaper to do it to open source.

[@tc2241]

He talks about that in the full video

[@Eagledelta3]

@@Kane0123 Maybe if it's found, but code and PR quality is quite often a lot lower in corporations than in Open Source. Not only is there a major timeline/deadline pressure that don't usually exist on the same level in Open Source, there is also generally a heavier amount of trust within the team and "LGTM" reviews are extremely common. An attack with this level of planning would be just as successful at a corporation as in Open Source.

[@drooplug]

I love how they worked on this exploit for a couple years, make it to the deployment stage, and an engineer that couldn't handle his code running a 1/2 second longer than expected completely unraveled all of their hard work and patience.

[@RaveYoda]

If a service requires ssh and is used by thousands ifg not millions of people and it's half a second slower then that adds up monitarily. I can imagine the need to optimize.

[@AlucardNoir]

Because you're not a corporation but an individual Linux user. Time is money, compute is money. Mony does not grow on trees. What for you is half a second for corporations it might be hours so o days of compute when it's all added up at the end of the year. You need to remember that it wasn't arch the system that was hit bit Debian buster, open use thumbleweed and the testing branches of other corporate distro's or distro's that see heavy server usage. This attack wasn't aimed at you and me but at big corporations with millions of dollars in hardware infrastructure running Linux.

[@yoshi314]

imagine how many more such efforts are going on right now.

[@thesenamesaretaken]

​@@RaveYodaIt's not that it's used by thousands of people, it's that it's used by thousands of professional autists. Corporations willingly waste resources in very many contexts.

[@JackDespero]

AI makes this thing even worse! There is an article about a guy who found that an AI hallucination was downloading a package that did not exist, so he made the package and he basically could do whatever he wanted with the thousands of people who downloaded his package unknowingly.

[@5467nick]

That's both alarming and hilarious.

[@TehKarmalizer]

I should start naming my projects by inciting AI to hallucinate.

[@AuthenticGadzooks]

That is genuinely so funny.

[@JimBob1937]

Rule number one of, proper, software development. Never blindly copy and paste code, you should understand that code and know the external libraries it references. If a person falls for this, they were already primed to fall for someone to exploit their bad code practices.

[@cornheadahh]

This is a success for open source software. It's often said that open source software is safe because there's always someone looking at the code to find exploits. Well someone looked at the code and found an exploit. Now the code is safer because of it.

[@mathijsfrank9268]

That is assuming that this is the first time something like this has been tried. It's more likely that this is just the first time something like this has been caught, especially how accidental the discovery was, which means all the other times it has not been caught. If you catch a shoplifter red handed, it's probably not the first time they've shoplifted.

[@khaoscero]

exactly. and jon doesnt think so. even tho you can infiltrate closed source just the same.

[@gwentarinokripperinolkjdsf683]

it took over a month

[@muskrat7312]

Right but paid state actors are doing this in proprietary code too. In addition the malicious xz code wasnt in the public github. So that should have been a warning to anyone careful.

[@ThisDaveAndThatJohn]

Let's keep in mind that the case of xz is quite isolated as it required multiple factors such as the original author who was kind of burnt out for years due to some long standing mental problems and out of desperation, he simply threw the project into someone else's hands only because it had been active for years and was promising. That's why we're all talking about it, it's pure novel. Don't get bamboozled by the survivorship bias.

[@likwidsage]

A more transparent process will always look worse from a security stand point. Closed source has the benefit of saying "oh woops...bug.". With open source you can tell what the intent is.

[@see-sharp]

No it won't! You should go back to college and re-study schools of "test-based security vs obfuscation-based security". The same way that a transparent process have more means of study to attack, they also have more means of repair and protection, and hands working on crisis situations. Go back to school.

[@doigt6590]

@@see-sharp I don't get your response. In what way are you in opposition to what OP says? To me it seems you two are saying the same thing.

[@YuRMoMiNHD]

@@doigt6590I assumed he was trolling

[@daze8410]

I think everyone is missing how quickly this was discovered. It's only been a month since the pr was merged. Would that speed of discovery be hindered by closed source? Absolutely.

[@MrVecheater]

​@@see-sharpI think by "look worse" he meant it seems worse rather than being worse

[@SimonM90]

It didn't take 2 years to catch Jia Tin. The malicious code was introduced by end of february. It did take days, or 2 weeks at most to catch it. Also he had to hide it in a binary blob. Hard to see how to do it in projects without binary blobs tbh I think the system is somehow more stable then people tend to think Edit: with npm its a different case though hahaha it gives me the chills to run some random packages

[@Waitwhat469]

1:36 This isn't a opensource problem. It's a software supply chain issue. People be acting like the alternative was hacking on abandonware binaries.

[@ElPikacupacabra]

There is no guarantee that closed source software doesn't have back-doors. In fact, given how many people Microsoft hires, it's much much easier to introduce the back-door into Windows. One data point is not sufficient for a conclusion.

[@ThePrimeTimeagen]

I do generally agree with this take

[@ForeverZer0]

EternalBlue (WannaCry), SolarWinds, and the list goes on... And here we are in an uproar because of an exploit that was on-par with them *almost* getting by. I understand that *some* bleeding-edge distros were actually effected, but it was caught, if even only by chance in the 11th hour, before it had a widespread effect. I imagine the long-term result is going to be positive, as people are now aware of attack vectors that we have taken for granted, and more vigilant of them.

[@AlucardNoir]

The NSA had over a TB of windows vulnerabilities they sat on untill someone absconded with then on a HDD and they imidiatly called MS to inform them of the vulnerabilities and to have them fixed before US infrastructure was attacked using their discoveries. Even if windows has been open sourced, most of the people capable of finding those bugs were already in the employ of government agencies looking to find bugs and vulnerabilities without and interest in disclosing then. And those that aren't work for the likes of palentir or actual security audit firms. Unless it's code you wrote running on an air gaped system you might as well asume it's been compromised, because if it hasnt, it soon will. Open source or not.

[@NElectronicSoul]

@@ThePrimeTimeagenThen what's with the bollox you were spouting in the vid then? Absolute L take indeed. "Dis guy predicted da futire, Opan Saus bad!"

[@dansanger5340]

"Much much easier" is a really bad take. Microsoft makes an effort to verify the identity and do a background check on their employees. They don't allow random, anonymous strangers to commit code to their codebase. The barrier at Microsoft is obviously higher, not lower. However, a good argument could be made that the closed source aspect could make it harder to discover malicious code after it is committed.

[@adjbutler]

1:30 xz was NOT infected for 2 years. It was Feb THIS YEAR.

[@kiosmallwood576]

Exactly. Primeagen is spreading misinformation now?

[@AtaGunZ]

yeah this dude is just blackpilled. They caught it before it made it to any major distro. It was a fucking nightly build that this made it to, and it got caught then and there.

This really wasn't a difficult prediction to make. It's been going on for a long time at this point. Remember the MIT people who tried to sneak a back door into the Linux kernel? Supply chain attacks on NPM packages? It's all the same thing, except this particular guy has been at it for a while it seems.

[@shankarsengupta1948]

The opensource community is learning the concept of defense against dependencies. "Just use a library" - has a cost folks.

[@no_name4796]

So let's just reinvent the wheel, shall we?

[@bonerjams2k3]

@no_name4796 calm down reddit

[@SaHaRaSquad]

@@no_name4796 Countless npm packages can be replaced by like 50 lines in a given project. Sometimes even 5 lines or less. And if you do that you take more control over your code, reduce the attack surface of your program and reduce build times & breakage, because those problems scale exponentially with the number and complexity of your dependencies. Stop whining about wheels and do your job - at least check the code of packages you're importing, you might learn a thing or two. Yes, dependencies have their purpose and make sense, but they're overused and people greatly underestimate their actual costs in the long run.

[@imranzero]

the "wheel" in software development has not been invented yet.

[@Sinci1]

​@@no_name4796 if you think about it, people reinvent the wheel constantly. Every sort of vehicle has different forms of wheels, a plane's wheel is not the same as a truck's wheel is not the same as a bike's wheel is not the same as a hoverboard wheel. Maybe it's more about not reinventing the core concept of a wheel, then it is about not having different implementations for different situations instead of having a one-size-fits-all solution for everything, just some food for thought, I dunno.

[@Elias_Ainsworth92]

A larger anything has more effort put into security. Company or open source. The difference is that with open source you can know how much is being put into security. Does Windows have more auditing than Linux? we don't know but hackathons would suggest not.

[@MoireFly]

You guys have actually worked at companies right? This idea that companies are _good_ at security is just so far removed from my experience. Companies do _not_ pay people for security; they pay people to create value; and preferably short term value. Security is _always_ a cost center in maintaining code bases. To the extent long term risks matter in practice it is almost solely either due to human due diligence despite incentives, or, because an organization's slacking off on a long term risk might be noticeable and _auditable_ in the short term and getting caught might really cause significant short term costs. Companies routinely do much less than the bare reasonably minimum because they know they're very unlikely to get caught, or if they are they'll suffer a slap on the wrist _most likely_ because they can plausibly claim everybody is just as vulnerable and security is impossible. The privacy of a corporate code overwhelmingly skews incentives towards cutting corners; managers do not and _cannot_ really measure diligence and security, and so if there's every _any_ kind of conflict between scheduling and security, there's always the incentive to cut corners because it's just not (sufficiently) measurable - and where it is measurable, those measurements turn into pretty kafkaesque security theater with an extremely slippery slope. This counts for corps of all sizes, including FAANG's; they may have more resources and a risk-reward bias that is a little more amenable towards spending at least a few dev hours on security and surely do better than average, but they're vulnerable due to OSS and other dependencies too (and have the scars to prove it), and spend microscopically to support those underlying projects despite that; i.e. they too are irrationally cutting corners because it's very hard to systematically tell waste from diligence. It's particularly amusing to hear this bit of cognitive dissonance from Blow, because he's a _prime_ example of the kind of thing large corps are surprisingly bad at despite the resource mismatch compared to OSS - he's a craftsman and chooses what he crafts because he understands it in a way a corporation with imperfect knowledge sharing between employees never can. What he does is exactly what corporations cannot due and that's why corporations spend untold billions on "security" and nevertheless have a terrible track record compared to the poor underfunded enthousiast in their garage. XZ's hack is notable also for its rarity _and note that even here - it very likely never worked!_ And let's not forget that this isn't strictly a dichotomy - (mostly) corporate OSS, certainly as long as it's truly a cooperation and not mere marketing - can exist and seems to work fine. OSS has problems for sure and I hope this close call causes changes in culture; but the idea that corporations maintaining secret code bases are somehow likely to do this better flies in the face of plainly observable security debacles virtually every day of the week. Corporations leveraging private secret code aren't just _also_ bad; they're obviously much _much_ worse at this than OSS.

[@tripnils7535]

This idea that there are 1000s of people constantly looking for security vulnerabilities in OSS is also delusional. In big open source projects, maybe a handful of contributers actually understand most of the scope to actually understand that there is a problem.

[@zesky6654]

​@tripnils7535 at least they are able to research possible vunerabilities, and there aren't incentives to hide the existance of problems.

[@tapwater424]

@@zesky6654 When was the last time you checked open source software for a vulnerability?

[@dansanger5340]

I worked at a large tech company, and there were definitely incentives to prioritize security. Admittedly, those incentives became more apparent after a series of security issues, but they do exist now. CEOs like to avoid the bad press, regulatory attention, potential lawsuits, and Wall Street displeasure associated with high profile security issues.

[@MoireFly]

@@dansanger5340 Exactly - they exist; but primarily indirectly as a way to avoid short term costs. It's a really poor fit for long term risk avoidance because it's often cheaper to mitigate the short term costs not (mostly) by mitigating the risks, but by hiding it, or by various tricks to limit liability, or by shifting legal responsibility as much as possible on those not making these decisions. And that's just the intentional ways to avoid paying for risk; I'm not sure the unintentional avoidance is any less worse in which risks get hidden not by intentional negligence but by the sum of lots of little social interactions; in essence: it's better for many people's careers not to "create" cost centers for the business by focusing attention and discussion on long term risks. People aren't stupid nor evil; but the incentive structure sure does encourage taking both intentional and unintentional risks, especially where costs are likely to be born by others. Even with the best intentions it's hardly surprising to see so many very well funded and capable firms suffer data breaches.

[@gmodrules123456789]

His prediction was that a backdoor would be clandestinely added and then activated at an opportune time with no way to close it. He was right about the backdoor, he was wrong about it being kept secret.

[@lowhat]

How do you know he was wrong. That assumes this exploit is the only one.

[@JohnCena-yu4mj]

1 found, 16 to go. The JB challenge started

[@gmodrules123456789]

@@lowhat For those not in the know, there have been other, smaller exploits that have been found and patched.

[@apmcx]

​@@gmodrules123456789 but you are still trying to prove a negative with positive data points. You can't just assume we find 100%

[@Spartan322]

In the xz case, it took weeks to discover by accident, I know Windows exploits that could backdoor you that resided in it for years that nobody fixed.

[@g3nko0]

But doesn't the same apply to proprietary software, how do we know that there's no backdoor in Facebook, Apple or MS apps?

[@jc_dogen]

how are randos committing code at Apple or ms

[@ZeProblematicz]

​@@jc_dogenit doesn't have to be randoms, if you want to achieve that you can either put your own people in those positions or try to flip/gain access to people in those positions.

[@autohmae]

That's my take too: this was a multi-year social engineering job on an open source project. This is the same kind of thing for example NSA does when they infiltrate a commercial company.

[@temari2860]

Easy, we don't need to wonder, we just know there are backdoors

[@g3nko0]

@@autohmae not taking into account how many accusations of leaving backdoors by previous employees of the companies happened in the last 10 years or might be even more (I've heard at least about Amazon, MS and multiple large US banks)

[@RemotHuman]

i mean they kind of did catch xz before it made it to non-beta production

[@Lazlo-os1pu]

Exactly… like how is this remotely a failure of open source compared to privately produced code

[@AuthenticGadzooks]

The code should have never been merged - doesn't matter that it was a test build. Finding the backdoor after the fact is not a win in my book.

[@nou712]

@@AuthenticGadzooks What's the alternative then? "Huhah! Just use windows XD!" Do you have a better alternative? Think about it, do you really trust an operating system where the government mandates backdoors? Windows security works on prayer(empty hope), and occasionally ducktape, and if you want to be uncharitable linux's security is like a kevlar vest, it might not stop everything coming at you but it's a whole lot better than just wishing you don't get hit.

[@CommanderRiker0]

By pure dumb luck. The stars aligned. Imagine how many are out there in open source undiscovered. I'm sure its more than 17....

[@tirushone6446]

John: "it's easy." Meanwhile the attack: one of the most unpresedented, novel, conniving exploits in recent history

[@cassiusbright1062]

That also took 2 years to implement and was thwarted in a month. Let's not forget that part.

[@MurtagBY]

It isn't THAT complicated

[@5467nick]

Someone somewhere made a quality-of-life improvement to an open-source program and submitted it to the developer. That they hid a backdoor in it isn't genius. The two years it took wasn't spent doing anything, it was spent waiting. They could have done dozens more submissions to other open-source programs in that time under different names and we might be none the wiser. If a single one of them goes unnoticed, that might be a problem. It's "novel" only in that people are surprised to see proof that someone had the gall to try it.

[@TheAxeForgetsTheTreeRemembers]

@@5467nick exactly. Plus, "unprecedented"... A multi-step attack isn't novel, it's called a strategy. You need to wake up if you think that kind of attack is rare, let alone novel.

[@squirlmy]

@@MurtagBY actually the code was that complicated, and maybe even novel. You have to separate out the social engineering attack and the actual code, and both were impressive, honestly. kzbin.info/www/bejne/gpK1fHytpayDirc

[@nextlifeonearth]

Thing with closed source is that these state actors are still there, as Jon said, but the community cannot even find the backdoor. This backdoor was found because it was open source, otherwise we would have never found it. Microsoft doesn't pay you to look for bugs, but to make them. They don't pay you to benchmark, but to change the unit the benchmarks use from milliseconds to minutes.

[@jeezusjr]

Also it would not have been found as easily if it did not cause a half second delay on ssh connection. Insanity.

[@4idenn]

Are we ignoring the guy who found it was actually a Microsoft researcher?

[@Lazlo-os1pu]

People conveniently forgetting the closed source FTX back door created the biggest financial crime in history

[@Ch40zz]

no, this bug was not found because it was open source initially... the way it has been found was by profiling, debugging, manual assembly analysis and then, finally, at the very end looking into the tar balls. this is actually much easier to find in the binary than by trying to find the point where the attacker started execution in the build process. open source in this case just made it easier to look at the initial steps in retrospect once it was discovered by the above profiling and debugging steps.

[@monad_tcp]

@@Ch40zzfinally, some who actually understands the truth of the matter, exploits need to be found at runtime, you can look at source and lose the causation chain very easily. People need to stop thinking that "eyeballs myth" makes things actually secure magically.

[@benderbg]

Guys don't use public roads, eventually someone is bound to have an accident.

[@no_name4796]

But that's different, for one big reason: trains exists! Or at least in europe, china and some other countries do it well enough

[@suddengun007]

@@no_name4796 in the US, trains have never had a problem with maintenance and safety..../s

[@SaHaRaSquad]

Public roads are built by companies and their paid workers. A better analogy would be eating random food you get from strangers without any food safety regulations and saying it's fine because in theory you could first check it in a lab.

[@no_name4796]

@suddengun007 but somehow in europe or any other country serious enough about trains, barely have any accidents. It's like having a fucking duopoly and the profit margins making decisions is a bad idea. So strange..

[@tedchirvasiu]

@@SaHaRaSquad it's fine because the random stranger would not be making food for free unless he was really passionate about it, and many people are eating their food so you would know if someone would get ill **copium off**

[@The1RandomFool]

The most interesting part of this to me is someone spent a few years and all that effort just to get caught immediately. If this were someone inside Microsoft and put something into Windows it could be undiscovered for a long time.

[@yoshi314]

they did not get caught immediately. this has been going on for few years. and they also merged a flawed commit into libarchive that was around for 3 YEARS. They only found it after the xz debacle, reverifying the malicious dev's commits.

[@The1RandomFool]

​@@yoshi314 The exploit was not actually active until 5.6.0 and 5.6.1. It wasn't active for 3 years.

[@CommanderRiker0]

Microsoft has an audit team, the XZ project apparently does not. A Microsoft employee found this by the way.

[@TheChillBison]

As novel as this attack was, it nearly succeeded (and others may yet) by exploiting the human factors. Pick an extremely widely-used low-level package with one maintainer and infrequent updates, use code vehicles that reviewers routinely gloss over, and use a system that has gotten comfortable with accepting collaboration from unvetted strangers that you never see or hear. I hope every maintainer of low-level system packages like xz are taking another look at commit history.

[@cemreomerayna463]

I am sorrry, but this is a horrible take. We are talking about one serious and very cleverly planned security breach involving years of trustbuilding and complex and layered planning. And it was found only because someone else was free to play with the code, test it, and catch it. Saying a no-brainer like there are malicious actors being motivated to attack OSS projects is not a new take. This one example attack vector that had a chance to happen for all these years happened once and got caught in short time without causing any significant damage (again, because someone could actually see the program). And neither making up the number of potential Linux kernel invulnerabilities from his 'backdoor" makes JBlow right about the current situation. He has extrapolated the overall security situation with a potential attack scenario that cannot happen in proprietary software by nature while dismissing all the other potential security issues with closed-source projects which has been realized orders of magnitude more. Is open-source software as secure as the FOSSnuts claim? No. Is it vulnerable to infiltration by ignorance or lack of motivation? Yes, absolutely. Is it still a much more secure development model than any model that closed-source software development can offer? Yes, by a large margin.

[@kiosmallwood576]

I agree

[@brentsaner]

I wouldn't be surprised if Windows has far *more* backdoors. macOS too. They are more expensive, sure. But they're also far, far, FAR HIGHER ROI. *Exponentially* so. The amount of checks in place in companies, even large companies, even *security-conscious* companies, does not outweigh the ROI value.

[@ABaumstumpf]

But that's the thing: They HAVE checks, tons of them, and tens of thousands if not more people whose goal is to find even the tiniest of exploits. Where as openSource... the library 5 repos down the stack of your fancy new framework got compromised 3 years ago cause there are only 2 unpaied people working on it and they can not even afford a dedicated build-server for integration-tests.

[@snooks5607]

@@ABaumstumpf depends highly on what type of company and product it is. the most security conscious company I worked at in past 25y ran a static analysis tool on the codebase once and recommended teams look into doing peer review, rest of them didn't give a rat's ass as long as the b2b customers paid their bills. could've backdoored everything during that time and they would not have found out. in open source at least if the project is included in a distros someone's going to look at the code even if the upstream maintainer doesn't (of course in this case wouldn't have helped since it was in buildscripts, that process could do with improvements and no doubt will after this)

[@brentsaner]

@@ABaumstumpf There are not tens of thousands of people in one company looking for exploits in any given product, much less all of them. Unless you're referring to "many eyes", in which case your argument is actually in favor of open source - you cannot view source code for commercial/proprietary code. Further, commercial/proprietary code *also* uses open source software, libraries, etc. Hell, Windows still has BSD networking stack code in it. Bloated dependency lists are a problem, but the lower in the stack you go, the smaller the dependencies tend to be because there are less corner/edge cases to cover, and less features needed. If webdevs stopped thinking they were systems programmers, this would be much less significant an issue -- I can isolate webservers. I can't isolate coreutils. But none of what you point out are even relevant to this attack in particular, which was a wetware trust-based attack first and foremost. Companies have a far lower requirement of trust to grant access. F/OSS requires months to years of steady contributions, if even, and access is typically granted incrementally. Companies require you passing the interview process. maybe 1-3 trial months. Then you get full access for your role.

[@brentsaner]

@@ABaumstumpfThis exploit got in *through* a check. It exploited the tests. Large corporations, even large *security-conscious* corporations, do NOT have "tens of thousands" reviewing for exploits. Unless you are making the argument for "many eyes", in which case you've made the argument for F/OSS, not commercial/proprietary - the latter is blackboxed, and thus source is not viewable. This attempt was caught precisely because the source was available. Do you think if a 500ms latency was introduced between versions of closed-source software, anything could be caught? Of course not, because the only people that have access to e.g. Windows are Microsoft and the extremely limited number of people who have signed the NDA (and even then, the NDA is typically scope-limited). You seem to think the entirety of F/OSS is managed by JS devs fresh out of bootcamp. This is false. Low-level F/OSS typically don't have marketing departments; a project's popularity is due to its stability and merit. It can take a project *years* of historical good reputation to be included in distro repositories. Worth noting Windows and macOS both heavily include external F/OSS code. Further, you ignore that this attack was *trust-based*. It took *years* of investment to even get it in, and the end result was years of investment burned because it only made its way into releases so new it was only in beta distro releases. Additionally, trust is granted incrementally over months and years of trust-building in F/OSS. On the other hand, what does one need to do to gain trust in commercial/proprietary? Be realistic - they just need to make it past the interview process. Maybe 1-3 months trial, probably not. After that? Full access to the scope of your role within the company. There is no personal vetting there; you are assumed to have already passed that threshold as part of the interviewing process. So again, it is likely easier to exploit human trust, which is exactly what happened in this exploit, in a corporate environment.

[@user-cz3sl5gr3n]

😂In my opinion, this isn't a very clever prediction. It boils down to "if humans collaborate, one might do something malicious". I mean, sure, but that's true of basically everything right. Eventually, a publisher will print a book with a hidden, offensive message in it that they didn't catch. Eventually, a government official will do something that is undemocratic, despite functioning in a democracy. Eventually, a twitch chatter will put some streamer's private information in their chat. I see these examples as equivalent to the prediction that "an open source project will have a malicious commit at some point and people won't catch it". I don't find any of the above to be particularly noteworthy predictions.

[@blenderpanzi]

The downloading random packages from a package manager wasn't what happened in this case, though. xz/liblzma is a well known program/library. It's not a random package and it isn't distributed via a package manager like npm or cargo. Linux distributions make an active decision to pull that in into their system. But yes, there was someone injecting a vulnerability in an open source project. That part he did call. However, on the point of downloading random packages: I read somewhere that people use AI to generate code, but then it generates imports for packages that don't exist. Except then someone actually created a package of that name and it got downloaded many times. yay Luckily in this case it was a security researcher, but could just as well have been a malicious actor. Come to think of it the logs of package registries of all the 404s of non-existing packages would probably be really interesting to malicious actors.

[@Bregylais]

Clearly, JBlow is JTan, and he would have used the backdoor to teach the entire IT community a lesson in cyber security by putting his KZbin-channel as a browser default page on every compromised machine on the world.

[@werren894]

jblows had nothing to do with this

[@unl0ck998]

Do you guys really think that there aren't state agents in big tech doing the same in closed source software?

[@boredbytrash]

Of course they do

[@AuthenticGadzooks]

I don't really see how that can be used as an excuse. The other guys are even more bad, therefor it's okay. Like what?

[@rusi6219]

And you have the same people pretending to be on the other side of the fence, just look at the amount of "exmilitary" and "former FBI/CIA" working "cybersecurity startups"

[@classyknight9513]

The whole point of open source was that you checked the source before you cloned and built the project, but that just isn't feasible in the modern day. It hasn't been in a long while, but people have relied on the hope that someone smart with enough time has checked the code for you and cared enough to find all the bugs. Which just isn't feasible, it never was.

[@jordanyates3349]

This is what worries me about Cargo. Such a “safe” language, but the community is churning out crates at a rate that rivals npm. A single rust binary might have 300 deps, how many of those are maintained? Vetted? Are the updates audited?

[@AuthenticGadzooks]

Absolutely agree with you. Everyone expects someone else to have done the due diligence already, but that probably isn't the case with most open-source projects. It's the blind leading the blind.

[@frankhaugen]

Intelligence Agencies are just as likely to plant code in private code. In windows and macos it's by law 😂

[@TC-cq7oc]

The "people can sneak into closed-source companies too" argument is missing the point. Yes, the Linux kernel is well-observed, yes SSH is well observed, but attackers aren't going after well-maintained, well-staffed open source projects that everyone knows about. The reason this attack (almost) worked is because it targeted a tiny repo with only one maintainer which nonetheless sat upstream of everything.

[@tonchozhelev]

I think if we wanna talk issues with OSS, that has to be the most important one. So many OSS libraries sit at the foundation of our modern technology and yet get no support whatsoever, and this is what causes issues. The whole CoreJS situation is a tragic example of this.

[@Pedro-cj9jm]

Agree - this is why supply chain needs to also be considered with these well observed projects.

[@prism223]

My take: Ironically, this situation shows the weaknesses and strengths of open source: * Some operator finessed a nerd into handing over his project. * Some nerd noticed that the project was broken. * The nerd heroes assembled to save the project. I agree with Prime's take that in the closed source world, this would have never happened and instead there simply would be a "front door".

[@Lars-ce4rd]

Dude made it 4 mins and 30 sec into a 20 min video and just wrapped it up right there. Dream SWE coworker.

[@engineeranonymous]

Ken Thompson is first "Reflections on Trusting Trust" ?

[@raymondhill7837]

This isn't what Relfections on Trusting Trust was about.

[@edugar88]

He actually credits the invention of that hack to an Air Force paper criticizing the security approaches in the MULTICS OS

[@engineeranonymous]

@@raymondhill7837 "...The moral is obvious. You can't trust code that you did not totally create yourself. (Especially code from companies that employ people like me.) No amount of source-level verification or scrutiny will protect you from using untrusted code. In demonstrating the possibility of this kind of attack, I picked on the C compiler. I could have picked on any program-handling program such as an assembler, a loader, or even hardware microcode. As the level of program gets lower, these bugs will be harder and harder to detect. A well-installed microcode bug will be almost impossible to detect. ..." The same point unless you write the code you can not trust it.

[@granitbajraktari1600]

@@edugar88 Which one is that?

[@edugar88]

@@granitbajraktari1600 See the acknowledgment footnote on that "Reflections on Trusting Trust" paper: "Acknowledgment. I first read of the possibility of such a Trojan horse in an Air Force critique [4] of the secu- rity of an early implementation of Multics. I cannot find a more specific reference to this document. I would appreciate it if anyone who can supply this reference would let me know."

[@davidspagnolo4870]

Genuinely really happy with the comments on this video. The whole idea of open source is that you cant trust code that you can't see. Thats why the attacker obfuscated the code in a test data blob. Then, even with the incredibly complex obfuscation scheme, someone still managed to find the exploit. If anything this whole thing shows open source working basically as intended.

[@Georgewilliamherbert]

Re US Government Back vs Front Doors... Remember the Elliptic Curve backdoor.

[@KennethFeur]

this attack vector should be called Jia Tan Blow

[@Parker8752]

It's worth remembering that the malicious code was detected primarily because the library was open source, while actually putting it in there was a process that took years. If a state sponsored actor wants to put the same type of code into Windows Server, they bribe/blackmail a sufficiently high up executive at Microsoft and Windows Server gets a back door.

[@ilearncode7365]

False dichotomy. You can have a projecy developed by paid developers for profit, and still be able to see the source code

[@DiSiBijo]

@@ilearncode7365 yep got him there

[@davidchristenes9062]

​@@ilearncode7365And everyone could have food everyday, "CAN" or "COULD" is just a abstract way of thinking about the world. Look at what happens and not what would happend or you think that can happen based in your ideology. Open source has exploits, they are founded and fixed most of the times. Private also has and sometimes they are not fixed until something big happens, that is the reality. Also we LITERALLY have the files thanks to Snowden that private owned companies deliberaly put malicious code inside it's devices, Dell had purposely put backdoors in the majority of it's devices to help US government spy endeavours, this is not something abstract, is somenthing that happened and was proved. Can be is almost anything my man, give up day dreams.

[@monad_tcp]

Being open source doesn't guarantee it wouldn't have malicious code, as we clearly see now. Being closed source doesn't guarantee the opposite either, that it will have malicious code. Those are orthogonal concerns. The problem is one of trust. Do you trust whoever is making the code, or not ? and do you trust their suppliers and the entire supply chain ? That's the problem of trusting trust. People think that the method of development (open-source/closed-source) or the licensing (free-software/commercial-software) is going to solve the problem of trust, but it isn't. The entire internet rely on basically the good will and trust of a very close knit of people, even for the BGP to work (which is absurdly insecure), and even for the certificate chains, which rely on companies (aka groups of people) not doing wrong shit (cof, cof, symantec). I hate this idea that people think of free-software (or communism) like a perfect system that's not fallible to all of the human deficiencies. I hate the uberman, bullshit idea.

[@rileyfletch]

@@monad_tcp But what is your point exactly? That closed-source software is better? Also the comparison to free software and communism is absurd to say the least. Regardless, I'd love to hear your argument for how transparently developed code could ever be worse-off security wise than closed-doors private industry developed software.

[@aniketbisht2823]

9:16 Linux is also developed in old school mafia style trust hierarchy. Linus trusts some handful of very close colleagues and they themselves take pull requests (and review them) from few other people they trust and so on. It's like a priority (search) tree.

[@Sharp931]

Linus as capo di tutti.

[@MrFreeGman]

I don't really get where the criticism of open source is coming from. It's working exactly as intended. If xz was some proprietary Apple or Google software, the exploit would never have been caught (or more likely would be an intended "feature"). Windows is most likely filled to the brim with backdoors of a similar nature. At least with open source someone finds it eventually, and this one didn't even take that long. It only affected people who use bleeding edge distros for like two months. Virtually no server or important infrastructure was affected.

[@pencilcheck]

Also the exploit consist of multi year social engineering to get around open source, the loop hole isn’t open source, but it is build CI trust library linking tarball without sha verification

[@pyrotek45]

the same issue applies to closed source. but in that case of close source software, less people can vet the code. so even with this, open source is still better imho.

[@SC1240]

i think that's a fair argument but there's also less people who can possibly commit malicious code. Hiring screens and properly managed read/write permission for code repos in an organization provide a layer of security on top of code reviews. Also less scope for most closed source projects. Still always a possibility of course

[@ABaumstumpf]

"less people can vet the code. so even with this, open source is still better imho." Yeah and it is the other way around: fewer people can even change the code, more people are checking it, and there is a whole slew of business-units that test, check and verify the code. "Hey bob, this newly added testcase triggered 5 warnings and 2 errors in our testsuite and failed all regression tests. go check out who did this crap". That's about how it happens.

[@eldrago19]

​@@SC1240Yeah. I go backwards and forward in whether OSS or proprietary is better against this kind of attack. I think this has put to bed the idea that all backdoors are spotted, but OSS is still more transparent even if the makers are less accountable.

[@pluto8404]

an attacker who can hide a backdoor in OSS can probably easily pass any Google coding interview.

[@ABaumstumpf]

@@pluto8404 "an attacker who can hide a backdoor in OSS can probably easily pass any Google coding interview" Considering that this code triggered SEVERAL compiler-warnings and was detected by multiple test-tools, as well as being highly suspicious to begin with... yeah no. Those PRs would all have been rejected and be ground for checking his work thoroughly. And that is not even to mention that it would fail even the simplest of integration-tests. But on a small openSourceProject maintained by a single person as a hobby with not test-framework? yeah there is no way he would spot that.

[@rmidifferent8906]

Ah yes, the legendary team of security specialists that are highly paid and highly motivated to make sure that there are no exploits in the company software that can focus only on that, because security of software is a priority for every company

[@rafaelbordoni516]

People keep talking about how elaborate the plan was, how they invested years into it yet they barely mention that it still failed. The most elaborate well thought out plan of what would have been the greatest hack in history failed against that so called "bad open source security". If they invested years into working at Google and tried to inject malicious code into Android would they have succeeded? Billion dollar companies like CD Project, Rockstar and EA got hacked so many times during the past 3 years, give me a break... The reason why they went after Linux is not because it's easier to attack, it's because it would have been the most impactful, 99% of all servers today run on Linux after all.

[@zimmerderek]

Minor accuracy thing that needs to be said: The xz exploit was deployed and almost discovered immediately. The social engineering part started 2 years ago, but there was no indication that "Jia Tan" was a malicious actor until the malware was uploaded. Additionally, reproducible builds would have negated this kind of attack.

[@MrRecorder1]

Just contradicting JB immediately here: the XZ-debacle also is kind of a WIN of open source resilience because it WAS audited. This whole thing was discovered in a month after it came out! It was released for sure, it was out in the open and blew up. Imagine how closed source would have dealt with that... would we ever heave heard about this? I will guarantee that we would not have, unless we pay for some enterprise-grade security BS...

[@AM-yk5yd]

>Imagine how closed source would have dealt with that Yeah. Easy. It would go like this: "Hey, Jim, Mr Totally Not State Actor, I've noticed that login takes too long and valgrind is going crazy. Can you look it up?" "Sure, pal, let me check it... Fixed it, now it should work faster" (and then he fix the bug in exploit without removing exploit) 'Hey, Jim, Mr Totally Not State Actor, it works fine now, thanks"

[@alexmipego]

The idea that the bigger the company has more security "auditing" is wrong. I worked for a startup (multi-million) from the USA, whose product is literal about cybersecurity, and every 2-3 months some hacker would ask for money because they found issues… even simple stuff like wordpress, and bugs like ignoring permissions completely went years without any insider finding it. Sure, maybe I got unlucky… but I've been doing this for a long time… so…

[@impromptu_ninja]

"Why did it take 2 years for someone to catch xz?" ... How long did it take to catch the problem with cab files? How long did it take to catch other compromised proprietary software? Would we even hear about it? Come on man... "How do we prevent" supply chain attacks? Vigilance. No matter what, no matter how. It's a wash on all fronts. Anomaly detection. Optimization becomes a fucking security feature. We're gonna have AI run the products in VMs and look for weird shit like cloud based antiviruses do except on gigasteroids.

[@AuthenticGadzooks]

And who exactly is going to be checking the code in all these repositories? Certainly not me and you. There doesn't exist some secret army of open-source cybersecurity engineers who are diligently checking every single commit on open-source projects. Open-source is the wild west of software development. This is the kind of blind faith that got us into this mess in the first place. "Someone else will figure it out" is not a solution.

[@erikp6614]

Well, ThePrimeagen, the xz/liblzma bug was found just some weeks after it was planted - the two binaries was uploaded to the repo 240309. Furthermore, the same problem is in closed source software, but we cant inspect the code. Count the number of intentional backdoors in closed source software/firmware that have been found - hard coded admin passwords etc.

[@PunnamarajVinayakTejas]

In fact, this attack was literally in closed source software. It was in a binary blob

[@janAkaliKilo]

To clarify: Prime is wrong here, it didn't take 2 years for backdoor to be found, it was the attacker who contributed legit code for 2 years to get commit rights. Backdoor was adde to the package ~1 month ago, and found by some guy reading the code (good luck with reading closed source). It also didn't make it to stable branches of Debian or Fedora (the two affected distros), so the damage is minimal.

[@MyTwinkie]

It wasn't found by reading the code. Backdoor was hidden and obfuscated inside binary blobs

[@janAkaliKilo]

@@MyTwinkie payload was in binary blob, but the exploit that modifies `make` (first stage) is in the tarball - you could open and read it. iirc it was in the ".m4" file.

[@raskolnikov6347]

he didn't find it by reading code, he found it because a performance benchmark was running longer than usual. The changes were hidden as @MyTwinkie mentioned

[@janAkaliKilo]

@@raskolnikov6347as I already said (yt removed the comment), the guy who found the exploit - noticed the delay and went to read source code of the program - there he found the suspicious code in the m4 bash script (first stage of the backdoor).

[@AM-yk5yd]

@@MyTwinkie Which he deobfuscated because he had the code that deobfuscated it. Freund didn't find exploit without source code. He found that code works longer than expected. If it was closed source, he would have stopped here as he is not RE by his own admission. But it was not, he then go on a mission to find out why it took longer than expected. And found the backdoor analyzing the code.

[@jrknsOFF]

"It's not gonna last that long," says the guy who doesn't even realize the sheer amount of great software he uses daily that's built with and relies on FOSS from the early days of the craft, let alone the many open-source protocols and licenses and initiatives that have allowed the field to progress as a whole, for everyone's benefit. Jonnathan really blows on this one.

[@CommanderRiker0]

Most large open source projects are corporate sponsored extensions now, so he is correct in a way. They are just shifting liability of their dev team from in house to "sponsored". Same software, no liability. At least when you buy software you can sue them.

[@aniketbisht2823]

Open Source needs to do better at code reviewing department. All of this makes "meanness" of Linus (Torvalds) all the more reasonable. The man is so dedicated that he goes out of his way to creatively insult the contributor (of bad code). I mean what would you rather have : no "political correctness" or xz like scenarios. When it comes to defect/bugs found in critical software, false positives may be "controversial" at worse but false negatives are absolutely dangerous.

[@fastflame200]

With outsourcing, it is even easier to Inject a team of Malicious Players by a Malicious (state) actor.

[@velho6298]

It's so funny when a dude asks explain it in terms of backetballs and Prime reads it as basketball

[@hanswoast7]

Auditable is not the same as being audited. I hope one day we have an everyday Joe OS that is fully audited by common sense and field experts. Not just by a compliance checklist or "trust me bro".

[@daniel29263]

People unironically think "AI" is the answer to all problem...

[@AM-yk5yd]

Which is horrendous take. I don't see how people actually can believe it. First it ignores the fact that attacker would have the access to the same AI and would make sure that it works around it. Just assume such AI exists and after 10 minutes it can report something. Now debian maintainers need to verify thousands of packages. They will not spend weeks prompting on XZ alone. State actor would. Second, static analyzers(PVS) are so much better. cURL uses them(plural). Third, runtime analyzers(Valgrind) are so much better. On of the reasons(beside login time) was valgrind, valgrind is why Freund's BS detector start ringing alarms. It had too many errors. And valgrind also was a reason why this exploit was already "found" before Freund. On hackernews there is a was already user who worked with the maintainer of the project to fix valgrind findings, that were introduced by the exploit

[@lezzbmm]

1:50 “u can’t prevent this from happening in the future” it’s so easy to prevent this from happening: u just need world peace + egality~ it’s easy

[@thepi]

Open source is not altruistic, but the same goes for proprietary software. Just because the approach to get something like this done is different does not mean backdoors can't exist in proprietary software. Especially smaller scale proprietary software, where there is no one to audit the code and no one to even notice the small little hints which malware leave behind. This comment is not about defending open source, or proprietary (god forbid). Ig this is just me saying that I have another reason to just rebuild my whole linux system with only the things that I know very well, maybe even freebsd idk. I can never be a 100% safe, but at least I can try to be *that* close to perfection.

[@StevenButlerplus]

The XZ exploit was in the wild for less than 2 months. That's kind of amazing. But you never know how many state actors exist in large commercial software projects, and you'd never know if a backdoor got put in there.

[@CommanderRiker0]

It was only caught by dumb luck, ironically from a Microsoft employee.

[@jeezusjr]

This is where CLEAN CODE comes in and we say NO to AWK decryption algorithms and especially autotools! Also, blow has an opinion about every god damn subject humanity can think of, and some of the time he is right.

[@NanoGamingGamer]

This was my first thought too when reading about it. Barely anyone can understand it without significant read time because of how obtuse it is.

[@RenderingUser]

Clean code is a myth. But this situation. This isn't just unclean code. This is obfuscated code

[@lukivan8]

Malicious code was in binary and run after 4 build time and test code steps. While still looking like compression algorithm even after getting this code which was encoded

[@Iscream4j0y]

I'm actually kinda captivated by the genius behind it, like a series of function calls that could have a bad data thing, but when all combined that does something else, or even on having a binary file that you replace bits of to turn into another thing entirely, I'm now just staring at my computer with shifty eyes... I've been a professional software developer for almost 7 years, at least me and my team are obsessed with hard-typing variables and return values wherever possible, and adding checks and tests to make sure things are acting EXACTLY the way I intend them.

[@tacokoneko]

11:00 Intel ME and other security coprocessors are the front door for the US govt

[@tacokoneko]

that is the unironic actual reason why china has to make their own CPUs for their national defense programs

[@notapplicable7292]

Number of fucking times ive named something wrong and its made it into production

[@potato9832]

Going closed source doesn't solve the problem. All software depends on 3rd party software. Instead of inserting bugs into open source, they will just insert bugs into dependent closed source. If they can't get access or get hired at those companies, then they'll build 3rd party software with latent bugs designed to be embedded in closed software.

[@maninalift]

I know it's easy to say this now, but it's not like we didn't all know this. We know that we all trust software that we can't possibly verify. We know that there are probably back doors in closed source and open source OSes alike. We just keep on going because we don't know what to do and it's easier to ignore it.

[@t3dotgg]

Not sure how I feel about this take. For every oss exploit like this, there are dozens of similar ones at private companies. With events like the state actors at Twitter and the chaos of Solar Winds, it’s hard to believe OSS is fundamentally insecure. I like the focus on incentives and how they differ in OSS and enterprise. I think the incentive to profit and grow at all costs is common enough to balance out any disadvantage OSS has. The insane nature of the XZ exploit isn’t a death sentence for open source. It shows how high the bar is to do such

[@ThePrimeTimeagen]

I keep going back and forth in my head on this one. I can't quite tell if it is or isn't more expensive. It most certainly is harder in a private corporation than open source, since I cannot just simply apply to work on the exact part of the exploit. It would be very difficult to get in that position of power and to get it in. But similarly, I could say the same thing for open source. You still have to build a lot of credibility

[@rmidifferent8906]

The whole argument is missing one most crucial point. Companies can willingly put backdoors into their closed source software, that then goes to be redistributed. In that scenario the community has no way of knowing or checking it, because they have no access to code. That skips the entire "malicious actor part"...

[@axelfoley133]

My response to Blow is that I'm less concerned about an individual adding a backdoor into a closed source projecet for their own selfish gain, and more about the company itself deliberately creating a backdoor for corporate gain. This backdoor was found in xz only because it was open source. Had it been closed source and instigated by the company that owned it, we'd never have known. What open source demonstrated is that it is possible to implement a backdoor into open source, but the amount of effort to obfuscate it is gargantuan, and they still got discovered. It's terrifying how far they got, but they still got caught.

[@eklipsed9254]

Johnathan Blow even predicted my birth

[@lukeemhigh]

Well, AKSHUALLY this could be preventable in the future, or at a very minimum the chances of this to happen again could be minimized, IF Linux distros would stop redistributing pre-cooked tarballs (or worse, patching them downstream with other libraries) AND/OR a standard is set in the build system of all major OSS repos. Hackers tend to target the "low hanging fruits", and all of the above points would clearly qualify as such. Standards and a bit of hygiene would drastically reduce the surface of attack. If it takes too much effort to pull off, the chances of something like this to happen again would be really low. The fact that this happened in an open source project means that we have the possibility to watch this situation evolve and observe the measures that are taken to prevent this type of attack from happening again. If this was Microsoft, Apple or something else, no one would have a clue.. BTW, it didn't took years to discover this backdoor. It took three months or so (which is still enough to cause some wreckage). This attack was planned and took years to pull off, but the actual commits of malicious code/binaries are months old, not years.

[@keyboard_g]

Run everything in sandboxes. Zones in Solaris/illumos was right. Bring back Unix.

[@themusicmanjph]

I'm still going to use Arch, btw

[@AM-yk5yd]

Helps that they were not affected.

[@olatrials]

I think there is a very important difference between what happened with xz and what Blow is talking about. With xz, yes there was a very sophisticated and thorough long-con at play, with building up trust and all that, but when the time came, a huge bomb was dropped into the source code, in a way that once noticed, malicious intent becomes extremely obvious. What Blow seems to be talking about is more subtle, where the actors over time build up potential exploits using a series of what is essentially fairly common mistakes. A buffer overflow, a hiccup in sanitization, missing null checks etc., building up to what can in combination become an exploit, and the "best" thing is that in this case, if done right, the actor has plausible deniability.

[@FeekOps]

we just proved the open source effectiveness. I understand his perspective, open source uncovers more issues than it generates though. If this was in Microsoft, we would never know about it. Open source lives by being out in the open.

[@tedchirvasiu]

Microsoft code is maintained by thousands of developers who work full time on reading and writing code for Microsoft. Unless Microsoft is willingly doing it, infiltrating someone at a company to pull this off is way harder than infiltrating an open source project with 1 burnt out maintainer.

[@FeekOps]

Fair point, you win. I am Unequivocally wrong

[@FeekOps]

@@tedchirvasiuthey lost their MSA key, they lost the keys to the kingdom…

[@tedchirvasiu]

@@SimonWoodburyForget There's one thing to get burnt out for free and there's another to be burnt out while you're full time employed and still expected contractually to do good work. If you abandon an open source project or take years to merge PRs or fix something there are no repercussions. If you want to infiltrate a corporation you first need to pass an interview, get hired with your real identity as a real person. You can't simply use a random username.

[@lucasbrant9856]

​@@SimonWoodburyForget lol my coworkers will murder my prs for a typo imagine if I cant explain some piece of code in there theres 0 chance my code is getting merged.

[@KevinNewman-qn7gc]

The idea that closed source software has a higher degree of QA is bonkers to me. With open source, everyone can see your homework, so you put in the extra effort. Most commercial software is something that passes whatever often under defined requirements their manager puts up, so they can go to lunch. It might work to pass the user test, but unless someone thought to add a security requirement, guaranteed it's not in there. There's just not enough time for a developer to do ANYTHING that isn't on the requirements list.

[@eggyparrot3844]

Open source is auditable, which is a great thing. The problem is that is is not auditED. Big difference

[@AM-yk5yd]

And exactly because auditable, the exploit was found. By auditing the code which wouldn't happen otherwise. Without source the only things Freund found was slow login time and tons of valgrind errors. This can be noticed in both closed and open source. Now going through tarball and performing all these `tr ` to extract blobs apply patches to the source code -- this requires source code

[@bearwolffish]

It's not just Open Source. Every major company has state sponsored actors working there. At least with Open Source you have hope of finding it.

[@gastonchevalet7342]

He uses windows.... Why should we listen to a windows using beta

[@diegofloor]

He didn't predict it, he GAVE THE IDEA! *dramatic chipmunk turns around*.

[@UnfiItered]

It didn't take 2 years, it took a few month. It's still bad but it's better than a closed source. If it was a closed source project, I'd bet no one would ever catch it.

[@AuthenticGadzooks]

If it was a closed-source project then in all likelyhood an exploit of this magnitude would have never made it past code review. Open-source made it possible to happen, but it also made it possible to fix.

[@noriller]

partly, it was good ol social engineering. Considering how fast they pushed for adoption, it was less about the "edging" and more about targeting whoever they wanted to target with that.

[@CEOofGameDev]

3:34 "No single human or AI would ever been able to spot it" Yeah, except of course the fucking guy who actually did spot it...

[@ky3ow]

i think he meant in code you won't spot it, they guy did spot it because ssh connection took longer than usual

[@CEOofGameDev]

@@ky3ow this changes nothing, the guy was just tipped of by the ssh connection slowness, there is a bazillion things that could make that kind of shit happen. But the guy somehow managed to track it back to the ONE package that had a backdoor in it, and figured out that, indeed, what was slowing shit down was a backdoor, without looking into the source code of the thing? C'mon man!

[@mback3713]

On the other hand... The Jia Tan group had to go through such impossibly complex hoops to try to obfuscate his exploit and spent ~2 years prepping... yet with open source, we were able to find this problem and shut it down within less than a month of when it was active and before it was in widespread use in distributions. Jia Tan is now called out and all his commits to open source are being evaluating -- and his group will be proscecuted if/when found. I believe there are many more back doors in closed source than open source... and... these are very difficult to find and probably impossible to prosecute.

[@droningonandon5589]

How do you know what's happening in closed source code either? Microsoft shipped with NSA backdoors throughout their history, at least with open source you do have that audit trail and ability to review the code. Where this broke down with xz was that they manipulated the process to include closed source binaries. Open source has a problem in that it is highly reliant on altruistic developers acting as gatekeepers, an often unpaid and under-appreciated workforce. But it still has major benefits over the alternatives.

[@ArkhKGB]

And in 1984, Ken Thompson already predicted it with his essay "Reflections on Trusting Trust". > The moral is obvious. You can't trust code that you did not totally create yourself. (Especially code from companies that employ people like me.) No amount of source-level verification or scrutiny will protect you from using untrusted code. In demonstrating the possibility of this kind of attack, I picked on the C compiler. > I could have picked on any program-handling program such as an assembler, a loader, or even hardware microcode. As the level of program gets lower, these bugs will be harder and harder to detect. A well-installed microcode bug will be almost impossible to detect

[@Omnifarious0]

1:33 - It wasn't in the code for two years. It was in the code barely long enough to make it into a couple of bleeding edge distributions. And it was found because some random person had access to the source and could audit it when he noticed some unusual behavior. So don't diss the person who tells you "because its auditable". How many back doors do you think Windows has?

[@cassiusbright1062]

But you see, a big company would never intentionally put backdoors into their software because... uh...

[@Sharp931]

The video is only 3 months old. GKH banned the University of Minnesota from lkml in 2021, because they tried to confirm basically the same theory for Linux kernel development, with a less sophisticated attack.

[@maximenadeau9453]

The wisdom of this man. Like splurging that web devs dont know how to code while streaming his editor with a series of nested if statements 5 levels deep.

[@kphaxx]

Nesting if statements 5 levels deep?! Everyone knows you gotta hide this complexity by moving your block of ifs the moment it goes deeper than two levels! Duhhhh what a newb JB is. Probably doesn't even clean code smh

[@meanmole3212]

@@kphaxx premature ejaculation is the root beer of all nested if statements.

[@SaHaRaSquad]

There are no nested if statements on his screen, stop lying.

[@minitbnn]

Caring about how code looks instead of how code executes is the reason web devs remain web devs.

[@evandroescutatrap131]

The guy of "Open source is audible" is right. We accidentally caught XZ. If it was closed source, we could never know and it may have passed. At the end of the day, yeah this type of shit will keep happening and we need that more people look into that stuff. I don't know how we'll do it and even if we find a way to make it super audible (like, really audible. Actual people looking into it, frequently) this shit will still happen from time to time. We shouldn't fear security flaws, not in a sense of "OMG WHAT WE GONNA DO BWAAAAAAAAAAAAAAAAAAAA", that's not the way to find solutions.

[@IgorGuerrero]

Hey Jon Blow, go audit windows then lol, the biggest clown on the internet.

[@4.0.4]

Jonathan Blow was super clever to even operate the Jia Tan account to prove the point. My hat's off to him. Point taken.

[@matkeyboard8054]

The best way to hide is in plain sight, confirm state level espionage

[@squisherderheld]

How is commercial closed source software not vulnerable against the same kind of attack? You can also figure out their supply chain and perform some social engineering to place people at the right positions. Just take a look at coporate jenkins pipelines and their complexity; with enough bad will and patience you can plant an exploit everywhere

[@AuthenticGadzooks]

Everything is possible, but it's less likely in a big company where there's actual oversight.