Log4J & JNDI Exploit: Why So Bad? - Computerphile

[@juliankandlhofer7553]

Honestly, I'm surprised it took this long to find a vulnerability like this. If you have userinput, it should best be treated like it was nuclear waste.

[@dsmyify]

Do you think this vulnerability was designed and placed into Java by an actor, or do you think this vulnerability was simply an accident?

[@blasttrash]

@@dsmyify I think log4j is open source and many jdk implementations are also open source, so I don't think someone would put this for malicious reason. Maybe it was just an accident. However what might be interesting is if this vulnerability is not something that was found just recently, but was known to bad actors since 2013 and they have been cleverly getting benefits out of this without letting the world know about this.

[@fisheatsyourhead]

@@dsmyify hanlon's razor makes me believe it was probably an oversight or mistake rather than a concious exploit

[@Sniperfuchs]

@@dsmyify This was almost guaranteed a case of "Wouldn't it be cool if this library could do X? I recently did something that would have benefited from this existing". And then it got implemented and activated as being on by default without thinking "Does this need to exist?", which is especially difficult in open source where there's not as much coordination.

[@ethan7930]

could be another eternalblue situation, maybe it was discovered a long time ago and was only leaked now

[@fernandopaul1]

JNDI is like Acrobat reader in early 2000s... 'Hey, let's make pdfs also play songs and make popcorn!' What could go wrong, right?

[@tyfyh622]

remember the times when excel didn't ask for confirmation to run a script?

[@petervansan1054]

@@tyfyh622 yes it would run it without it, I still remember macro viruses

[@veggiet2009]

I... I want a pdf that can make popcorn

[@vincei4252]

ActiveX was da bomb.

[@LarixusSnydes]

Actually, the issue was the extension of the PDF format with the JavaScript abomination that had access to your file system. This is made worse if you used the win9x/ winNT/XP default that any user you addded had administrative rights from the start... If you want interactivity, just create a different file format, so that you can clearly tell them apart and avoid the interactive one.

[@AaronSherman]

The key issue is that this wasn't a bug. It was the accrual of a large stack of features that no one was evaluating the implications of.

[@fwiffo]

It's a bad and unintended behavior of a piece of software. That's a bug, just a complex one. Just because it's caused by a combination of a bunch of intended features doesn't negate the fact that it's unintended behavior. If this was the *intended* result, we should call it a back door.

[@Unit_00]

@@fwiffo I personally prefer to call it an exploit. Exploit -> unintended application of intended functionality (every module is doing what it's supposed to) Bug -> unintended functionality (at least one module isn't doing what it's supposed to) modules being functions, classes, etc EDIT: An exploit is a design mistake, a bug is an implementation mistake.

[@Stierguy1]

@@fwiffo A bug would be unintentional. This is just incompetent software design.

[@charksey]

I would agree with you up until the "send and run arbitrary java objects," that is inexcusable. It's perfectly fine and valid to send POJOs that the other system has loaded in the JVM, but it would have been better to stick with XML or JSON or some such. Running any kind of code that somebody (¿over the internet!?) has handed you is absolutely horrific.

[@BigDaddyWes]

The semantics don't really matter.

[@oliverbatt3559]

"Do one thing and do it well" and "Keep it simple, stupid" - two core tenants of the Unix philosophy.

[@ChristopherGray00]

Just saying a philosophy made in the absolute infancy of software and microprocessor computing doesn't necessarily apply to modern day. The vulnerability doesn't exist because it had the functionality, it existed because of extreme and utter negligence of the developers working on it, the vulnerability existed for months and wasn't fixed.

[@Amine-gz7gq]

@@ChristopherGray00 modern day software engineering is a horrible mess. you must be really a fool to appreciate it (just like taking a jab from a corrupt industry and obeying the government).

[@ChristopherGray00]

@@Amine-gz7gq It sounds like you are really out of touch with software engineering in general, using modern software while not knowing it. Linux and BSD unix has been exploding in features, stability and security in the last 3 years, the amount of work on the open source software world is more than it has ever been in history, saying "modern day software engineering is a horrible mess" is very ignorant.

[@Amine-gz7gq]

@@ChristopherGray00 OK enjoy your life.

[@seriouscat2231]

The word you are looking for is 'tenet' (a belief), not 'tenant' (a rentee).

[@devnol]

Now that's the explanation we were looking for. Thanks once again to all the folks over at computerphile!

[@steveroger4570]

19:28 "build library that do one thing well, and then compose other things on top of that if you need" is one of the most underrated aspect of programming these days. A lot of programmers these days just love to bundle too many thing into one component. And not just library, having too many responsibilities in one component just generally make thing worse.

[@bluescanfly1981]

Right, we're so adverse to tradeoffs we'd ideally prefer not to have to think about them, let alone impose them consciously and deliberately. For example, in typical corporate settings developers don't get to discuss architecture choices and tradeoffs until the existing architecture is decided inadequate - again, most often ,not by the developers. At this point, a hectic modernization effort begins - but you cannot modernize stuff all at once, so you get bits and pieces patched in - leaving you with the worse of both architectures.

[@seriouscat2231]

Every ABC should include XYZ, because every alternative to ABC also does. "Your framework does not include a toaster? Ha! I will rather use one that does."

[@AsaLeighton]

Agreed but also keep in mind too that programmers and whole teams are under the constant pressure to meet deadlines and production levels. ANY software that offers a boost in productivity, and becomes popular, because if it, becomes a target for hackers and malicious actors of all levels. FOSS products and their libraries can be accessed and updated by almost anyone and some are actively pursued to achieve malicious ends. Even without malicious intentions bloating of components is something that is likely to continue and create these situations.

[@tiagotiagot]

Imagine how many serious vulnerabilities like this are hidden in closed source apps...

[@ChristopherGray00]

Not many that are used to any great extent really, this is a case of extreme negligence.

[@tiagotiagot]

@@ChristopherGray00 That we know of

[@Nors2Ka]

Fewer because in closed source and especially paid apps someone is actually responsible for the code instead of the open source mess where nobody is responsible and nobody is getting paid to maintain high quality.

[@tiagotiagot]

@@Nors2Ka I guess you're not familiar with Windows, Flash, and so on...

[@bitterlemonboy]

@@Nors2Ka You're very wrong about open source software. Closed source has way more bugs and exploits.

[@superjimnz]

This doesn't seem like a bug, rather a fundamental design flaw. It seems very unclear why this vulnerability wasn't obvious from the outset.

[@bimjean1053]

every bug is a design flaw

[@ChristopherGray00]

@@bimjean1053 false, there's easy mistakes and there's extreme negligence, log4j's situation was the latter, it had the functionally to remotely execute code, the vulnerability existed for months, and it was not fixed in that time.

[@Nothing-f8z]

@@bimjean1053 Form my understanding every part of this vulnerability is working as intended.

[@Pavium]

@@Nothing-f8z Yep. Every individual piece is doing exactly what it was meant to. Someone just linked them together in a way that was short-sighted from the security standpoint.

[@mcspud]

Well, since everyone here seems to be an "expert" I humbly request you dedicate your valuable time to supporting this Apache project and other free services.

[@warmachineuk]

I'm a Java programmer and use Log4j2. I know what JNDI is but I wasn't aware Log4j2 did that and would not normally want a logger to do that. As far as I'm concerned, if I want my logs to include environment information, my code should look that up itself and tell the logger. Dynamic lookup by the logger should need it to be switched on in config, which will also include permitted server names.

[@davidmorton8170]

I'm just stunned that anyone even thought that was a good idea to begin with. To quote Emperor's New Groove: "Why do we even have that lever?"

[@jeffreyblack666]

@@davidmorton8170 The bad idea is doing anything to user input before sanitising it. If you want to go down the path of why have the logger do that, then why bother with a logger at all? Just make your own. create the strings in your own code and send it to the appropriate place to log it, without using any library.

[@OMGclueless]

@@jeffreyblack666 It turns out this is a bad way to do security. There have been hundreds of exploits written against code that looks like this. For example: Sanitizer #1 looks at a string and sees that it has no HTML tags in it and is HTML-safe. Sanitizer #2 sees it has invalid unicode in it and normalizes it all to valid unicode. Now the string is considered "safe" and printed into an HTML file, but it turns out that Sanitizer #2 allowed a malicious user to sneak some HTML into the input in a way that undid Sanitizer #1's job. The consensus best way is to try and represent the user input as faithfully as possible and do the sanitizing as *late* as possible. For example, use an HTML template language that sanitizes strings automatically when it includes them, or a JSON library that automatically encodes unicode with escape codes when it serializes it. Strings are *never* safe unless a programmer controlled them the whole way. A log library with a structured API that treats strings as unsafe user data by default is the safest way to log data. Concatenating strings in your own code is the least-safe way.

[@dtkedtyjrtyj]

@@jeffreyblack666 Yup. Add a timestamp, then to stdout with you. It's someone elses problem now. _Every_ feature described is something I don't want from a logger. Variable interpolation? No, I'll just build the string myself. Recursive interpolation? Wut. Why? I want logging to be _fast_ not add potentialy _everything_ to it. Global variables. No. Never even once. Network requests as a result of logging? You're joking right? I'm not saying I wouldn't have added anything like that; but in retrospect they all seem insane. It's a logger; what should it do? Log; faster is better. It shouldn't be an entire application taking a life of its own.

[@warmachineuk]

@@dtkedtyjrtyj There are features I need from a logger more than just writing to stdout with a timestamp: per-category logging level configured at run-time; simultaneous writing to standard output and log file; automated archiving and purging of daily logs.

[@Yokovich_]

This has consumed my life for the last 2 weeks...

[@andysmason]

Same here. I am not even a cyber guy, but a developer/integrator that answers to their scans.

[@geoffblanduk]

14:00 Just for clarity, SLF4J is _not_ LOG4J. SLF4J is the Simple Logging Facade for Java - it is a facade that can wrap any logging library; such as logback, log4j V1 (V1 is not affected by this vulnerability) or indeed log4j V2. In fact SLF4J helps here as it makes it easy to swap in a "safe" logging library without having to change any code at all (although configuration files will need changing).

[@EwanMarshall]

To be fair, one can also turn of JNDI lookups in config file for log4j quite easily to limit the issue too.

[@stewartdahamman]

Does this vulnerability relate to log4j-core only or all log4j installs of the versions in question?

[@EwanMarshall]

@@stewartdahamman From the Log4j homepage: Apache Log4j2 versions 2.0-alpha1 through 2.16.0 did not protect from uncontrolled recursion from self-referential lookups.

[@stewartdahamman]

@@EwanMarshall Thank you for this 👍

[@elodens4]

There's something wrong with java culture when you have a widely used facade package just for logging. People aren't joking about the AbstractSingletonAspectInstanceProxyFactoryBean

[@arpitdas4263]

Appreciate the wonderful professors for explaining it as succinctly as possible

[@TheApiasecki]

Seriously you guys break these complex topics down so well. I look to you first when trying to understand something..thanks for this video, definitely cleared up some of the confusion I had with this bug/feature.

From what I have read about the exploit, as long as variable expansion was limited to configuration files (to log hostname, etc.), it was safe because configuration files are trusted data. It is only when log4j2 started doing variable expansion in the logged message, it started to be a problem (and would be some problem even without JDNI, though much, much less of a problem), because logged message is untrusted data - as you said, you often want to log what the client did. The problem with composing libraries out of many small pieces is that all those layers of indirection are not free; they have some performance impact, and for logging this may be a problem. About the closing remarks: FreeDOS still exists (as open-source project), and is still being at least somewhat actively developed.

[@thewiirocks]

As they mentioned in the video, Log4J is already doing the composition of the libraries under the covers. So there’d be no performance impact by exposing it. Quite the opposite. Most developers would NOT compose it and performance would be faster.

[@EwanMarshall]

It isn't the expansion that is the issue, but the recursion of it. Expanding is fine, but do not expand the data that returns.

[@iabervon]

The first problem was that it was configured by default to do lookups in messages. They disabled that by default and later made it impossible. The second problem was that, after doing a round of lookups in your template (which is a trusted string the programmer put in) it did additional lookups in the result, which now contains untrusted data.

[@tlniec]

Thanks for this - I was curious about this vulnerability and hoped that it might merit a Computerphile video (you've set the bar so high, it's hard for me to sit through most other computing-related channels' videos).

[@eainen]

something I wanted to say is that in part this is the beauty of open source software. if there is a vulnerability in closed source code you are dependent on that maintainer to fix it, that is if you are even made aware of it in the first place. with open source like you said, you can fix it yourself

[@jeromethiel4323]

Say it with me: "Code Bloat" or "Feature Creep." Coders need to create tight code that does exactly what it says on the tin. A logger should just log. A math library should just do math. But people keep shoe horning in features that aren't needed for the core functionality. As it was said, make simple tools that can be picked over and choose the features you need. When you get kitchen sink software, you may or may not know what was included in that kitchen sink.

[@Stierguy1]

Stop inventing languages in languages. I'm looking at you, format strings.

[@RonJohn63]

But the whole point of OOP is adding new features via inheritance. And don't forget running agents on other servers...

[@absalomdraconis]

@@RonJohn63 : Neither of those should be treated as the purpose of OOP, it should be treated as just a way to use encapsulation to create programming entities that act in coherent, self-consistent ways. As the most famous of the multiple Unix principles says- Do one thing, and do it well. Don't try to produce honey with your composite structure just because it inherits from the "hexagonal grid" type that gets used to implement honey comb.

[@izayus11]

I mean sure. Yes. But at the same time this is very much a non answer. This is like saying, "we don't need to work thru the local and global economical, political and social transformations needed to bring about the change of the world. All we need to do is. Be nice. To other people.

[@HiddenWindshield]

@@izayus11 That analogy doesn't make any sense. @Jerome Thiel's advice isn't going to prevent *all* security vulnerabilities of course, but they would be a lot fewer and a lot less serious if developers would take that to heart.

[@Anthony-wg9lu]

"are you sitting comfortably , then I'll begin ..." . classic.. thanks guys really enjoy the way you do content

[@thatcreole9913]

I would love to go back and take more classes with these guys as the profs.

[@drgr33nUK]

You can with an Oreilly subscription. Mike runs some courses there.

[@thatcreole9913]

@@drgr33nUK 🤩🤩🤩 my hero!!!

[@WistrelChianti]

I want to super like this comment

[@coderider3022]

Nice guys yes but aren’t they a Java shop ?

[@hrclful]

It's great when you could check your Open Source code for any vulnerabilities. Have you ever tried to `npm install` any package? At the end you have a hundred dependent packages you're responsible for.

[@nkazimulojudgement3583]

Lol

[@arwahsapi]

Remember the day when "left-pad" library almost broke the internet?

[@Luxalpa]

It's not as much because of bad developers as it is because of a bad standard library though. JavaScript lacks most features required to write code, which is why they are being imported.

[@jeremypnet]

18:40 this is exactly it. The logger should never alter the messages to be logged in any way. If I ask the logger to log something, I want to see exactly that in the log file, not some derivative with fancy substitutions. If I want the Java version, I’ll figure it out and put it in the message before sending it to the logger. This is an egregious bug in log4j 2 even without the JNDI issue.

[@gaptastic]

Love computerphile, just love the thoughtful responses of what's happening now. Also, I'm still waiting to hear the one-man band guy (based on instruments in background) play us a tune in the key of computerphile.

[@Computerphile]

Hah don't hold your breath :) Sean

[@ywanHK]

Cybersecurity experts getting ready for Christmas Log4j: *And I took that personally*

[@amirgamil]

Quality video as always 👍 Gotta admire that beautiful piece of machinery in Steve's background 😻

[@CommodoreHorrible]

If you heard an applause-like noise in the distance earlier this month, it was the sound of every infosec analyst collectively facepalming

[@SpringDivers]

Outstanding presentation, gentlemen. Thanks you and Merry Christmas.

[@asdfghyter]

18:20 This is exactly the same problem as with heartbleed in openssl. The library implemented an obscure functionality that almost no-one used, but it was enabled for every single user by default.

[@IllidanS4]

One thing I've learnt the hard way is to NEVER mix format strings with user input. Imagine if "%s" in something like printf actually recognized the % characters in the actual argument, well Log4j does this essentially, so...

[@vikramkrishnan6414]

Exactly, surprised it doesn't escape strings

[@CottonInDerTube]

@@vikramkrishnan6414 You mean we need prepared statements for logging? :D Well, ... could be a solution ... or we just decide that loggers only log given strings. Hm .... :)

[@MrBenAttenborough]

Never realised how much of a cool dude Sean is. Look at all those guitars and sick drum kit. Very nice!!

[@brujua7]

Loved the casual chatty vibe. A 2 hr podcast of these three would be a banger!

[@rikschaaf]

People should indeed modularize their libraries more. Log4j should put resolving of JNDI calls in a separate module and JNDI should modularize their capabilities, like approaching LDAP servers or the ability to retrieve back something other than clear text. If a programmer doesn't need/want it (or doesn't even know it is a feature), they should be able to do without.

[@JR-mk6ow]

True, just cut the complexity

[@ThisNameIsBanned]

Exactly. Turn stuff "off" by default or not include it in the first place. Have people activate whatever features they want to use, so they invest at least a little bit into learning what the feature is doing in the first place.

[@omri9325]

That's the reason java open source libraries are piled with vulnerabilities

[@othernicksweretaken]

Maybe such advanced or fancy features which the majority of library users won't need should have been switched off by default and would require those few who covetted to make use of them to have a little extra effort by explicitly enabling some sort of macro switch and having to recompile it altogether?

[@jeffreyblack666]

@@ThisNameIsBanned Sure, why not go all out. Disable every single method and property and so on, requiring some kind of config option to turn it on. That way they need to find the config option, and turn it on to know what it does. That way it can be nice and secure.

[@kwicklaunch]

So basically ${...} is an embedded eval(...) which can cause problems on its own. When coupled with JNDI to a bad server allows for the creation of a REPL or shell.

[@thewiirocks]

Bingo

[@vokuheila]

A logging library that can make arbitrary network calls while logging. What could go wrong?

[@nlac73]

based on possibly unfiltered user data:)

[@TheGiantHog]

Thank you for this, I had the log4j issue at work but had no idea what the vulnerability actually meant was possible in our APIs

[@Snarkbar]

I love how Minecraft is where people first discovered this bug. At least they used it to mess with each other there instead of taking hospitals offline or something.

[@TheRavenMad]

Not true unfortunately. Apache credited the disclosure to Alibaba's cloud security team.

[@pmcgee003]

@@TheRavenMad Whoever else ALSO discovered it, it WAS clearly discovered by some Minecraft players.

[@JCBOOMog]

On the anarchy minecraft server 2b2t one person exploited it and did bad stuff

[@siddharthkhamithkar5920]

@@JCBOOMog fitmc fan?

[@JCBOOMog]

@@siddharthkhamithkar5920 I've watched him ye

[@HamzaHafeez7292]

Its so amazing how much impact one person can have. I mean, one day you see him win Man of the Match for Real Madrid and on the coming Sunday, you see him demonstrate the vulnerabilities of Log4J. You have got to love this man.

[@Lantalia]

The main problem is that it does this recursively, so even if you know about the substitutions, you can't actually do the careful sql injection protection of only passing raw data via substitution into fixed strings

[@petervansan1054]

I am glad we are still on log4j 1 :)

[@tepelem]

Glad it ended on a positive. This video just confirmed every doubt I have every time I download a package or a 3rd party software. It’s the Wild West. Compounded by the pressures companies put on their developers, why wouldn’t you go for the quickest easiest option, if your boss is hanging over your shoulder.

[@idontwantahandlethough]

Yup, this kinda thing seems like it's essentially inevitable. A lot of managers effectively incentivize devs to put out hasty, sloppy code... so of course that's exactly what they do (because why wouldn't they?). This is what happens when you let non-tech people (specifically: business majors -_-) manage tech people ;)

[@JoolsUK]

Its also the reason App development has rocketed so fast and the digital world has become our world we depend on so fast. Cyber Security has been the afterthought.

[@strifef7]

was waiting for the chaps at computerphile to get this out! Thank You!

[@CristianConsonni]

The only criticism that I have with this video is the reasoning at 24:08 . If MS-DOS was open source it may still be maintained if there were some uses for it. There is plenty of projects that have died because of lack of maintenance or because they have been discontinued in favor of entirely new ones.

[@fiver-hoo]

FreeDOS exists, it is the FOSS replacement for MS-DOS, is still actively maintained and has it's uses to this day.

[@nutday5]

I've been waiting for the explanation from you guys on this. GJ, thx. I can't do anything more now, I have to go, have a Merry Christmas.

[@DShadowWolf]

Note: MS _has_ published the source code for MS-DOS -- but long before that (in the 90's, in fact) the FreeDOS project was born and is still in active development. It even has long file name and internet support - dven a graphical web browser. So... The statement about MS-DOS is somewhat correct, just lacking a bit of knowledge.

[@jeffspaulding9834]

I don't agree with his sentiment at all. Maintained software is "past its sell-by date" when it is no longer useful to anyone. DOS is useful for a variety of things - hence why FreeDOS has an active user community. It offers something that few other operating systems do, in that you have full access to the hardware. Does he not realize how much COBOL there is in the wild? Or how many engineering firms still use Fortran?

[@gdclemo]

If people want to maintain obsolete operating systems as a hobby, good for them. It's their free time, who are we to tell them what to do with it?

[@jonathanguthrie9368]

@@jeffspaulding9834 I don't agree with the sentiment either. No one is doing anything wrong by maintaining software, however old, that people still use for things. On the other hand, using software that isn't maintained is probably not the best of all possible ideas.

[@TheRealRedRooster]

Yeah, mentioning DOS here is a bit like blowing how air here. There simply is no attack angle here like it applies to JNDI and Log4j in DOS. Any DOS. Even having a 3rd party web browser like in the case of FreeDOS doesn't change that...

[@jeffspaulding9834]

@@jonathanguthrie9368 Yeah, unmaintained software is always a bit of a minefield, and it's only going to get worse. Here's an example: I use a lot of 90s era software for working with industrial equipment. And it's fine, for the most part - although some of it won't run on 64 bit Windows and has to run in a VM. None of it listens to the network by default (unless you spin up a simulator, and even then I'm behind the corporate firewall or air-gapped). Today's software is often written as Windows services with a thin shell program to interact with the service. As that software becomes unmaintained (long before the hardware for it gets replaced), it'll still be installed and running in the background on the machines of every technician and system integrator that still needs to work on that hardware. Ticking time bombs - can we say Stuxnet 2.0?

[@bluesmasterelf]

Amazing video. I've shared this with my company's security chat.

[@jacobstamm]

“I’m finding myself in the situation, which is my fault really, where I’m using a library that I don’t really understand” Node developers: “Allow me to introduce myself”

[@Nainara32]

I mean, this is basically everyone. Understanding every nuance of every nested dependency in a modern application is an impossibly high bar.

[@aravindpallippara1577]

Especially with the looming deadlines over our heads

[@BigCarso]

Yeh that's a pretty throw away line that's not very useful

[@TheAudioCGMan]

hm do I also have to understand the runtime env and the operating system?

[@yuvalne]

I was waiting for this one. Thanks!

[@dvol]

I don't think the problem is that log4j does both logging and formatting -- those two things go together more often than not! It's that the API doesn't give you a good way to separate data and control -- those go in the exact same string. So yes, very Bobby Tables. Flogger is an example of a Java logging library that does both logging and formatting, but in a safer, faster way, a lot like using query parameters in SQL.

[@amyshaw893]

my two favourite computerphiles in one video!

[@HisMajesty99]

The past week has been a nightmare at work due to this issue

[@LuminousWatcher]

I completely agree with the moral of this video: When you include a library, you are not only reusing code (and maybe "following best ppractices"), but you are also getting a lot of features and bugs that you will have to handle in some way. I am in the process of modernizing a program, and decided to go with some modern frameworks. suddenly I am spending much more time patching than ever before. However, this doesn't mean that my original code was better, it means that now I have all the included libraries' patches being flagged in some databases and some tools like npm and githubs vulnerability scanner picking them up. I would like to proprose that the take away point be expanded to be: Be concious of every decision you make regaringg your programming. As a modern programmer you might not need to be as aware of memory or disk usage as when the progcessors were 8-bit, but the exposed interface and potential consequences of breaches and vulnerabilities means that you have to think ahead. You might intend your software to be used in one way - but can it be (mis)used in another.

[@hilly_billy9531]

The JNDI warning in stdout are turning into a nightmare on pyspark data pipelines.

[@codeman99-dev]

Oh man! I'm so glad I left that world. I already disliked that employer enough. I couldn't imagine having to patch pyspark on GCP.

[@bentationfunkiloglio]

Fantastic explanation of a complicated topic. Well done!

[@SimGunther]

_When a logging library does more than logging_

[@ktxed]

unix philosophy violation

[@doctorscoot]

That JNDI ‘feature’ is used in the JEE specification to obtain ConnectionFactory objects to connect to remote servers.

[@fwiffo]

That's an insane design pattern.

[@thewiirocks]

@@fwiffo No it’s not. The purpose is that you can configure an application server and code will look up key features (e.g. database connections, ejbs, message servers, service handlers, etc.) at runtime. It makes code highly portable so that you don’t have to recompile your artifact for each environment. (Which IS insanity.) The real WTF is that a logging library is doing JNDI lookups based on log messages. Recursively. 🤦‍♂️🤦‍♂️🤦‍♂️

[@devilaverage6718]

A really breaindead design, especially without whitelisted servers.

[@doctorscoot]

@@devilaverage6718 yeah, and if your network design includes allowing all your production servers/containers to hit JNDI requests externally there's probably no stopping tons of other vulnerabilities as well

[@fwiffo]

@@thewiirocks It is insane. You're trusting another computer to give you arbitrary code and assuming it's safe to execute. And by default, it assumes every computer in the world gives you safe code. That's fundamentally broken. Maybe if it works off of a whitelist it's OK, but it is wildly unsafe by default.

[@DanGBaxter]

Just before Steve mentioned it, I was thinking myself how many of the libraries I use in android are built (moshi/retrofit adapters come to mind). Have a base lib and have extras that you have to implement manually. On top of the bugs argument, it can make the library much smaller. It's not a huge deal for a dev to add an extra lib in a gradle file.

[@quistian]

Great survey and analysis. Thanks.

[@MCRuCr]

I'm a software developer and my boss (a physicist) now thinks bc of log4j that open source software is _fundamentally_ unsafe... God working in big companies sucks I wanna develop indie games or something

[@traister101]

Ah yes my favorite security by obscurity

[@Ubya_]

It was safe BECAUSE it was opensource, imagine if the sourcecode wasn't public and you had to wait for a company to fix it for you. It might have been ok, but usually thousands of heads are better than a few with extreme deadlines and a maybe not optimal work environment

[@grill-surf-bust]

"I can't imagine anyone solving a problem except under penalty of being fired."

[@krishnabirla16]

Waited so long for this.

[@Spanner0jjm]

One patch is to just remove the JNDI functionality from log4j, I had heard that there was a worm created that went around using the exploit to go around patching wherever it could!

[@boenrobot]

A worm that patches the vulnerability that it entered through... That sounds genius... If true.

[@gameconner]

I think you are referring to 'Cybereason Log4Shell Vaccine' use the exploit to patch the system.

[@fwiffo]

This was attempted for the code red exploit (with some unintended consequences). Also, some 2b2t hackers were doing this to patch log4j vulnerable Minecraft clients.

[@ketsuekikumori9145]

"Exploit, bug, feature... pick your term." Everyday consumer: exploit. IT technician: bug. Hacker: feature.

[@Stierguy1]

Software Engineer: feature

[@bluegizmo1983]

What's scary is that this RCE is so incredibly easy to do, and it's attacking something that is so widely used, you just know people had to have been using it for years without people knowing it! This kind of RCE is literally a once in a decade kind of find. Log4J is ran on so many things, it is impossible that they will all get patched, so this RCE will be valid for a long time to come.

[@DanB-0]

Today jfrog found a log4j-like vulerability in the java based H2 database console.

[@WilliamAndrea]

I'm surprised you didn't bring up "left-pad"! It was an NPM package that one guy published, then took it down cause he and NPM got into a spat, and it ended up breaking a ton of other packages that depended on it.

[@crtxl]

Nice touch: seeing the Atari monitor for most of the video reminding me of the time when my Ataris (800xl and 1040ST) were my only computers. No Internet or script kiddies. Full postal address displayed in demos without privacy concern. And so on.

[@taneliharkonen2463]

I was eyeing out for this video! Thank you! :D You are now the official vulnerability explaining channel in utube! xD :)

[@AdamFJH]

@16:46 the better question is why would you use a third party logger when the one Java provides does an excellent job if you just want to log strings to a file or a console. I've always used the one java provides because it's simple and there is less code for me to do security audit on. EDIT: @17:13 that's another major reason why I prefer to use the libraries that come from Java's SDK API oppose to third party libraries.

[@maagiapall]

Does the default one allow logging to DB?

[@AdamFJH]

@@maagiapall It doesn't have out of the box support for logging into a database. If you want to log into database using the default one you'll need to write a Handler object to be used for the logger.

[@Knifewolf]

Very informative! Thanks for going over this.

[@DampeS8N]

This is why the cavalier attitude most modern web devs (and other devs) have about using package managers and pulling in a million libraries from a thousand different sources _terrifies_ me. How many more of these are out there? The standard argument goes "Why should I build something when I can use an open source library maintained by hundreds of people? It'll be more secure than I ever could make it!" and there is truth to that, but you can _only_ do that if you _know everything the library does by default and you stay up on it_ which implies to me that you still should use as few libraries as you can. There is no perfect solution. There is no way to escape having to be the master of your environment. It is just as irresponsible to pull in any old library as it is to code without concern for security. You have to vet code, no matter who wrote it.

[@bonononchev634]

I assure you, those devs don't like it one bit, but when management is presented with the bill (a.k.a. effort estimate) nobody actually wants to pay it to be done in a safer way...

[@eekee6034]

@@bonononchev634 You must associate with sane devs. I got out of Linux distro development when Linux desktop software started developing circuitous and deep dependency chains. It was also pretty obvious that Linux package managers made the wrong things easy. They make it very easy for devs to use libraries while hardly helping system builders to check the quality of those libraries at all. After my experiences with Linux, I almost got depressed when I learned package managers were appearing for programming languages. Ever since then, I've been thankful I don't have to work with any of this.

[@DampeS8N]

@@bonononchev634 I wish this was true, but unfortunately I have met many the React/Angular dev that sees absolutely no issue with pulling down a wad of spaghetti on the order of hundreds of megs. A Lovecraftian horror, knotted into Lolth's web in the form of node.js code.

[@idontwantahandlethough]

I'm really, really hoping that we can get computers to review code before we mess things up too badly. I'm not sure how possible that is, or if it could ever be a full replacement for a human (surely eventually, but how soon?), but it would definitely help. I feel like most managers who haven't written much code themselves don't quite understand how important it is (and unfortunately, they're also the least likely to listen to that input in the first place), so it'd be a lot easier to get them to pay for an AI to do it rather than "paying people to sit around staring at an already finished product" (an actual quote from an especially bad manager I had).

[@eekee6034]

@@idontwantahandlethough I hope so too. Everyone likes to quote the halting problem to say "it can't be done," but I like to point to the fact that the halting problem applies to computers without finite limits. ;) I'm not smart enough to go deeper than that, though. There is ongoing research around finite computers being able to understand themselves in ways unbounded Turing machines can't, but the last I heard was it's all blue-sky mathematical work. The Agda programming language is part of it.

[@thelegalsystem]

Not even remotely on topic, but that's a lovely window that Mike is sitting next to.

[@bakslashr]

The default should have been to sanitize data being logged. This really is not much different from an SQL injection attack.

[@--Nath--]

Keep in mind logging is like breathing - it happens everywhere at all times.. so every bit of extra code you whack into the critical path - you create more work. Log4j (like any software) is juggling competing non functional requirements - performance, simplicity, security, extensibility etc etc.. along with functional requirements. It does raise the idea that logging should have different categories of log message - trusted/internal only stuff and messages that may have some part that comes from the outside world (because rarely do you log a single value like that - log4j is quite often a String that is stuck together via various things, which is a prime case of injection risk.. Anyhow - this has been an important realisation for everyone to think more carefully about logging as a vector for attack.. or look more closely at what log4j settings should be disabled to lock it down more. Ironically one of the best tools for detecting and analysing security attacks is the logs! So it is a critical capability. I also look on the bright side: this will mean many enterprise systems get an upgrade!

[@bakslashr]

@@--Nath-- The moment they decided to start evaluating ${} in the data to be logged, they added more to be added to the "critical path" To prevent injection attacks, or manipulation of the data being logged, the data coming from users needs to be sanitized, which is likely going to be on par, or less than, with evaluating the nested ${} in the first place.

[@--Nath--]

@@bakslashr in just about every enterprise system out there or framework that shields from directly calling log4j - that needs to happen further up the call.. by the time log4j gets it it's too late (you've got valid substitution value stuff mixed in with a string with bits of user input shoved in). The log4j calling JNDI bit is the real problem though - that feature needs to be gone. Substitution values are kinda key to how logging works. Anyhow - I think this situation, while a massive a shitstorm does mean people will be thinking more critically about the matter of logging.

[@kxjx]

@@--Nath-- I don't think this is quite correct? Log4j is executing as code a string that the developer though was going to be treated as data. Its gonna be extremely difficult for a dev to sanitize this from their log inputs if they don't even know the functionality is there?

[@--Nath--]

@@kxjx developers have *assumed* which is the problem.. Everyone uses log4j by default almost but without really looking at the functionality.. (because hey, it's just logging.. right?) and why this has been so widespread a problem because it is so common/humdrum an activity that no one paid much attention to it. Logging message/communication payloads are one common problem I try to educate people about.. but this one extends to any field that gets logged.. so log4j-injection is now something everyone is (hopefully) going to be aware of..

[@konstantinkh]

One high profile example is the way this has hit Minecraft community. By default, a Minecraft server will log chat messages using log4j, which created just about the simplest way to run this exploit on an enormous number of machines. There are fixes in place now, but at the same time, it's hard to estimate how many MC servers are still running old versions of the code with the old configs.

[@BurnabyAlex]

24:30 Actually, FreeDOS is being maintained today because there's a need for 'DOS' style machines.

[@Raletia]

Re: around 24:10 mins. There is an open source version of DOS, called freedos. I know it got used by companies for things like bootable bios update usbs and such as recently as the mid 2010s. That's not it's original purpose though, from what I remember it was meant to be a replacement for MS-DOS with support for more modern things like usb, long file names, etc, and if I recall, let you run DOS applications with strict memory requirements easier. It's been years since I looked so I could be mistaken on some things. Even if it's 'past it's shelf life' I'm glad it exists, it helps make older hardware useful still.

[@ThisNameIsBanned]

Turns out, features should be "off" by default and people need to turn them on if they actually want to use them and hopefully understand what its doing. That way problematic features would not be "on" by default, which is troubling in general. Classic "Poka-yoke" , make the system so making mistakes is difficult and not normal.

[@tackline]

"Secure by default" as it is known. That has been Oracle's policy for many years.

[@leftaroundabout]

‘features should be "off" by default’ - Yes, but OTOH this can just make the problems more obscure and even less likely to be found. If it's merely off by default it'll be all too easy to turn it on based on some tutorial and then forget about it. A better takeaway is that such problematic features should just _not be implemented_ at all. It should never be possible that a logging library executes arbitrary code based on string inputs.

[@fennecbesixdouze1794]

Log4j itself is off by default. You have to install it as an external dependency. The fact that so many people are using Log4j proves how little people pay attention to security considerations of even adding external dependencies, let alone just changing a config.

[@jeffreyblack666]

Features being "off by default" just leads to loads of confusion with people trying something and not knowing why it doesn't work until they find whatever config option they need to change to enable it. What you are suggesting basically amounts to having no functionality unless you explicitly enable it. Imagine if there was a config option for every possible function.

[@dealloc]

Like the Vulkan API... But I agree. Things that aren't necessarily useful in majority of cases should be off. Sane defaults over simplicity for some things.

[@nathansnow]

Another thing to consider here is that even if you aren't using LOG4J and think you're safe, you might very well be using a plug-in or have a dependency your code relies on which uses LOG4J. So just like how we inherit dependency bugs, we also inherit security vulnerabilities

[@Verrisin]

I definitely never expected log4j to be able to do JNDI calls.... it just feels absurd.

[@ASU885]

I've been waiting for this video :)

[@aiistyt]

In twenty years of java I’ve never worked on a project that didn’t use log4j

[@AntiKarios]

Great video! Thanks for explaining so well.

[@efraim6960]

it is still beyond me that a logger is given this kind of functionality lol

[@Korgolord]

Great video!

[@farrongoth6712]

24:22 I think this is the wrong way round I think from personal observation and some experience that open-source allows software to die a lot more than close-source projects, and it makes sense significant amounts of monetary investment goes into proprietary software, telling investor we need to do it all again is not always an option. Saying that I'm not entirely certain it's a closed/open source problem, but rather who and how many users the software has and what portions of the user base uses what features. Also there are communities that maintain ms-dos even though it's proprietary, mostly for posterity as you say, and mostly for gaming.

[@tylerbreisacher5841]

yeah, i was thinking if MS-DOS were still actively maintained it would look a lot more modern

[@rushyscoper1651]

yea i think the real issue is using other ppl code that u don't understand or know. it kinda easy to make a library that is useful for other and he can leave a bug there for u to exploit, it ok to use code without vetting for personal use but for professional use no that just lazy and a habit that need to be stop specially for big respected companies.

[@leftaroundabout]

I agree. With a closed source tool, once it is widely adopted you'll usually have some companies ending up relying on it for decades and pumping money into that somebody keeps maintaining it. If it's open source, they can just hire somebody to port the crucial features they need to a new platform.

[@andreasklindt7144]

@@tylerbreisacher5841 FreeDOS, last release come out a week ago.

[@oifbert]

Thanks for covering this topic. I think a little bit more simple logging code examples would have helped explain the concrete error scenario. For example logging a username obtained from a web from could have been a great example for a "bad" logging pattern since the user controls the input there.

[@brynyard]

The problem we got is the people who added this feature without thinking of the consequences!

[@yavuz425]

nice, been waiting for the opinion of you guys on this.

[@darknewt9959]

It's a shame that industry is littered with end-of-life applications which are business-critical and only work with some legacy Apache/Java stack. This vuln will be hanging around for years.

[@sau002]

.NET developer here. Excellent explanation. Very insightful.

[@JoolsUK]

Steves on point here. Its crazy imagining all the Government agencies pulling apart global open source libraries for RCE vulnerabilities to be used for nefarious purposes and yet the digital world is using these miltifaceted open source libraries.

[@Anenome5]

I used to think hacking scenarios like in Ghost in the Shell were unrealistic, but the more complex chips and code get the more hacking opportunities seem to be appearing.

[@arrrda__]

I love Dr Pound

[@FhargaZ]

0:56 Nice Atari combo you got there. I really liked the explanation, great job. When we used a cool python library for an asigment I thought about something like this situation.

[@nickeshchauhan5661]

I disagree somewhat with Steve's enthusiasm that anyone can fix open source code. In principle that's true, but it overlooks the fact that people need to be willing, need to have time, need the skills to understand the code, need the ability to fix it, and the character to do it for free. In practice, it turns out that Venn diagram is very small. Some of these libraries can be a hundred thousand lines long and connect like a giant spiders web, and the larger they are the smaller that Venn diagram gets. I support open source, but it's not the magical cure that many devs believe it is.

[@miltiadiskoutsokeras9189]

True. Not every mechanic can repair any machine.

[@kimgysen10]

I hate open source. First of all people take for granted that other people work for free; second it's unreliable and third there is no quality control.

[@BigCarso]

@@kimgysen10 There is in fact probably a lot more quality control that code you write yourself. Brand new code is very likely to have more bugs than a library that has been battle tested for 10 years

[@miltiadiskoutsokeras9189]

@@kimgysen10 nobody is forcing anybody to work for free, many people work paid tasks on open-source code, you are missing the point. Open source allows peer review which has proven superior to internal QA. Most large open source projects are funded by large corporations. Most secure software in existence is open source. Proprietary things are very limited in scope of review and usage compared to open source. There is no contest, proprietary code is worse in every way but still has its fair share. Even closed source products are based in open source software these days.

[@miltiadiskoutsokeras9189]

Actually the problems on OSS are starting to come up when the motives for correctness and quality are replaced by competition for market share. Look at Linux vs OpenBSD: Linux wants to be everywhere, OpenBSD wants to be correct. Their security status is a gulf. Researchers managed to get malware into the Linux kernel upstream. That cannot happen intentionally in OpenBSD as of now.

[@davidmeijer937]

Great video, as always!

[@FUNktshnl]

If people were using log4j like Mike in his example via slf4j, this wouldn't be a great deal, since slf4j allows to swap out loggers in an instance. It just provides the generic logging interface for the programmer that calls an actual logger. Doesn't matter which one it is.

[@MikkoRantalainen]

This is a great example of "defaults matter". The fact that Log4j was easy to use and it enabled remote JDNI support *by default* nearly all software using Log4j is vulnerable.

[@camofelix]

Unfortunately, about an hour before this came out, an equivalently critical vulnerability was discovered for Lua in apache server that allows for a classical buffer overflow attack

[@edwardoyh2550]

"Someone hacks Mike's server, it's not that hard because Mike wrote it" had me 😂😂😂😂😂😂😂

[@spoddie]

Q; Why is a logging library 250,000 lines? A: Java

[@erathornzyltris]

The brings back what Steve Gibson from Security Now often says: "What could go possibly wrong?"

[@ijabbott63]

24:20 Not MS-DOS, but there is still a community of people maintaining its open-source equivalent, FreeDOS.

[@jongeduard]

19:13 Most important lesson, same with printf %s indeed: You should simply never accept any kind of variable input for parameters which can be expanded by some template engine or which can otherwise be executed in some way. So if all input can be recursively expanded (like with log4j), you can/should never ever process any variable input, making it practically unusable. :P