Log4j Lookups in Depth // Log4Shell CVE-2021-44228 - Part 2

In this video we dig a layer deeper into Log4j. We get a quick overview how Log4j is parsing lookup strings and find the functions used in WAF bypasses. Then we bridge the gap to format string vulnerabilities and figure out why the noLookups mitigation has flaws.
Part 1 - Hackers vs. Developers // CVE-2021-44228 Log4Shell: • Log4j Vulnerability (L...
My lamest GitHub repo ever: github.com/LiveOverflow/log4s...
--
00:00 - Intro
00:38 - Chapter #1: Log4j Lookups in Depth Debugging
03:50 - Log Layout Formatters
06:56 - Chapter #2: Secure Software Design
09:21 - Chapter #3: Format String Vulnerabilities
13:58 - Chapter #4: noLookups Mitigation
15:15 - Final Worlds
15:42 - Outro
=[ ❤️ Support ]=
→ per Video: / liveoverflow
→ per Month: / @liveoverflow
=[ 🐕 Social ]=
→ Twitter: / liveoverflow
→ Instagram: / liveoverflow
→ Blog: liveoverflow.com/
→ Subreddit: / liveoverflow
→ Facebook: / liveoverflow

[@jeffvalore5010]

your "secure design" section hits the nail on the head. They made the classic mistake of trusting user input, and never considered "how could this be abused?". JNDI aside, it shouldn't have taken a whole lot of thought to consider that anyone who has access to read the logs but not access the server or code directly, can just log all the server env vars (DB connection strings? AWS creds?) and potentially steal information. IMO this entire substitution "feature" of log4j never should have existed. Even if it was "secure" this is still a lot of bloat for features that a very small subset of log4j users would ever utilize.

[@theyreMineralsMarie]

Exactly, this is a classic separation of responsibilities problem. No logging framework needs to also have this level of environment lookup functionality.

[@Lukas12220]

Or disable the feature by default

[@mskiptr]

Yep, this is a job for plain variables and lambdas and not a separate string evaluation engine

[@TheCardil]

Log4j did nothing wrong, apart from having to much features. The culprit here are developers who pass user controlled data to be logged. That should never be done. Those values belong only to the internal database, and shouldn't be leaked to logs. If devs wouldn't log user data, it would be safe to get logs from production. Often that's not possible as they are littered with user owned data.

[@hb-man]

@@TheCardil Logging uncontrollable user data is what logging is about: "Could not login user 'attackerstring', wrong password." You cannot expect the developer to escape for any special chars to prevent code injection, just because the logging engine has surprising features.

[@monognome9202]

As a developer, this vulnerability caused so much chaos throughout the weekend in webapps using Java. But you explaining it makes it seem so simple. Amazing video 🙌🏽🔥

[@jamess1787]

I'm still astounded at how much is still written in Java. It's wild.

[@monognome9202]

A whole lot man. It blows my mind everyday.

[@rjhornsby]

I’ve always liked your sort of casual handwritten visual style, and it especially shines here. I can more or less read Java, but there’s so much code in just this one tiny - almost incidental - library, it’s nuts. The way you approach highlighting, focus in on small code blocks, and your red/white callouts make it super easy to follow your narration. So, thanks! and Merry Christmas.

[@Fardeus]

true.....

[@MuhammadBinZafar1]

I'm glad that you have 380+ videos. I never want to run out of your videos. I've been listening since 2018🎈. Always enjoyed every single moment with you. Each video is unique, knowledgeable, perfect 🎉. I wish the best for you!

[@AFchump78]

“I have no clue about Java or maven with dependencies” i felt that in my soul. I went back to 3 deserialization issues after the log4j learning curves to reattempt that exploit after the Java and java development learning curve that log4j caused. Beautiful and tiring at the same time.

[@vaisakhkm783]

😂i trained a complete year in Java and I made many projects in my training period,. But after getting job I never touched it... Now when I tried to make a hlowld , I even forgot to put public static void main in class.... ....😑 All the hard work just vanished innu head... Now I wondering how I did build big spring projects that time...

[@spicybaguette7706]

I'm curious whether anyone ever has needed the _recursive_ lookups... It's a logging library! Not a string substitution one! Why is this even enabled by default in the first place?

[@lgfrbcsgo]

Because someone somewhere thought it would be a good idea to be able to change the log format on the fly without a redeploy?

[@ritamdey8427]

This was my reaction when I read about this feature! The lookup and complex formatting should've been a add-on library rather than the main one, which should've stuck with logging strings and simple formatting like %t etc.

[@karigucio]

for real, never would have thought a logging library somewhere in the code checks for infinite recursion

[@desmond-hawkins]

The point is to avoid crashes due to the recursion limit being reached, not to explicitly log recursively. It's common to log the details of some objects with toString(), and sometimes people will log objects that have toString methods that are more complex than they should be. To be clear, none of this is about an ideal case, but to prevent a _worse_ case to an already sub-par situation. If you log an object with toString which causes it to dump a whole bunch of information to try to be helpful, and this information is calling methods… you can easily get into one of these situations. Imagine a Circle class, with a center of type Point and a radius of type float. You could have an overzealous toString do something like: return "Circle(center="+center+", radius="+radius+", area="+this.area(); - now if your call to area() does something like log.debug("computing Circle area for " + this); - then you end up blowing up the stack. Purely by accident.

[@cdhagen]

This library is a prime example of over-engineering! That's why I don't 100% agree with people who claim that it's just „because a few overworked guys are thanklessly maintaining this thing for free“. Yes, maybe, but who forces them to build crazy features like this into what *should* be a relatively straightforward logging library?

[@lal12]

No I don't agree. Yes the jndi stuff and java indirections are examples of overengineering and insecure practices. However the bug in specific is basically a classic injection bug, which should be a common pattern how to handle this.

[@cdhagen]

@@lal12 True, unless you are overwhelmed by your own complexity. ;-) Also, maybe this library would have more maintainers if it weren't so complex. The fact that this "classic injection bug" has gone unnoticed for so many years speaks volumes.

[@lal12]

@@cdhagen well I don't disagree. And Log4j probably has issues with documentation and usage patterns. That being said it is kind of unfair to blame the developers of Log4j instead of the developers of the individual applications.

[@DanKaschel]

They're not slaves to you or anyone. They're allowed to add features. Even ones with bugs. They define what it "should" be, not you.

[@cdhagen]

@@DanKaschel Of course, I can't bar them from adding some more of these golden "features" where a logging library makes random calls over the internet depending on what magic string is passend in. Keep up the good work, I guess! 🤡

[@superblaubeere27]

Some people take it a bit too far with the OOP. Splitting code in three-liner functions and douzens of classes does not really help the readability of the code haha

[@vert3x560]

ppl just don't understand OOP. They make 3 liner functions, and then some monstrosity 500 lines of code and for-ifs. To be honest. Clean Code clearly says: Keep It Simple. So I can imagine rewriting log4j into some clean representation with almost all features in like no more than 10 classes. One class one feature.

[@DanKaschel]

Good OOP would have prevented this issue. There never should have been multiple places where the nolookup needed to be checked for whether to evaluate a lookup.

[@santi5655]

7:40 totally agree, it's crazy how complex a logging library can be for no reason at all, simplicity is also good for keeping the code maintainable and easy to understand

[@T0m1s]

It's a mindset. Java developers seem to want to split their code into as many classes as possible, and this behavior seems to persist even when they switch languages. OOP is cancer.

[@klotzklotz1]

I think another interesting aspect of this vulnerability is that, if your organization used "defense in depth" and configured firewalls to block outgoing LDAP connections, exploitation of the vulnerability does not easily work. Unfortunately many people do not restrict outgoing network connections of their servers though :-(

[@berndeckenfels]

Yes egress filtering, not for “outgoing ldap” but “outgoing anything” since the attack allows arbitrary tcp ports.

[@klotzklotz1]

@@berndeckenfels ok, if the target port of the outgoing TCP connection is not necessary the LDAP port I stand corrected :-)

[@EER0000]

Great video, nice to see an actual dive into the code and not just more talking heads. I wish it was easy to keep the codebases clean and simple, but so far any project that I worked on with a team devolved in a mess of dependencies. The hardest part about keeping things simple, is to convince stakeholders that it's worth spending hours to implement (e.g.) centralized logging yourself when there's a library like this already available. They'd rather see me working on a shiny new button than on stuff that nobody sees in the backend.

[@passerby4507]

But you can't just write everything in-house. How do you decide what gives?

[@EER0000]

@@passerby4507 when I was a kid I would be making everything myself using just notepad, because that was the way to flex as a dev. With time and experience you learn which kinds of things are worth doing yourself, and which ones are not. I would never write a pdf converter for example, but I have written a basic SOAP parsing library using standard xml functions.

[@VivekYadav-ds8oz]

No wonder it took so much time to detect this vuln. Most people who just wanted to explore the library were probably intimidated away by how bloat it is. Actual function that does the job has 5-10 functions of bloat over it. Can't believe so much CPU time has been wasted on just Java OOP things throughout the computing history.

[@telnobynoyator_6183]

not as much cpu time as you might think because java optimizes that away later in the vm, it's more of a readability issue

[@luis_musik]

@@telnobynoyator_6183 i think you're overestimating how much the compiler can actually optimize away. most oop features (especially stuff like virtual functions etc) are not zero cost.

[@telnobynoyator_6183]

@@luis_musik The java compiler doesn't optimize those out, but the JVM almost surely does when translating the java bytecode into binary.

[@DelusionalLogic]

@@telnobynoyator_6183 "surely"

[@telnobynoyator_6183]

@@DelusionalLogic In a sure way

[@jackbendtsen8002]

Absolutely, simple code means all of the code that your program could run, not just the code you wrote. I feel like not enough people truly understand this. Thank you for your insights beyond the topic at hand.

[@tunasalmon1243]

The best gift for christmas for an aspiring developer. Thank you so much

[@NareshKommuri]

Thank you for sharing, really interesting perspective. Awesome in-depth analysis in few minutes. Cheers!! 👏

[@hblaub]

Really nice Java library. I think we should load custom code per socket from somewhere, too. It's going to be totally safe and sane.

[@kawcco]

This library is hilariously bloated. Who knows how many other vulnerabilities are hiding in the Log4J Jungle?

[@TenderBug]

What terrifies me is how many Apache libraries out there. Even more so other opensource libraries with big community driven projects. Log4J ! Serving Lookups Since 2013

[@kawcco]

@@TenderBug Agreed. We need to extensive code review and fuzzing of the entire Apache family so that we can cover our bases.

[@jhbonarius]

Have you read many comparible libraries? In my experience al mature libraries are like that. It's both a result from feature creep and common design patterns. The thing is: everybody's now complaining and nitpicking on this one, as something was found. ( a captain hindsight moment. )

[@kawcco]

@@jhbonarius The fact that they're all like this *is* bad! I'm not defending **anyone** who writes code like this!

[@jhbonarius]

@@kawcco is it your responsibility to defend them? Anyhow, code usually doesn't start out "bad", but 'evolves' into it over time. Call it "code rot". Especially in understaffed projects this can happen, and log4j was understaffed and open source. This is by far not the worst code I've seen. And considering the popularity of log4j, it wasn't disfunctional.

[@nekuskus5218]

Thanks for the great explanations! Merry Christmas!

[@einsjannis]

I really enjoyed your videos about log4j I really happy that some one added something new and I actually wad able to learn sth new

[@mateowang6570]

Very very interesting from a developer perspective. Thank you for this!

[@anacierdem]

Having the developer pass the contents instead of parsing various things is a much better approach even from a software development viewpoint here. It is inversion of control, which effectively reduces inner complexity.

[@anttijokipii]

Thanks for you content. It is really good. I have worked as programmer/architect/CEO on multiple decades, but have background from hacking. It is interesting that security and development is actually so separated.

[@caujka]

Lovely explanation! Absolutely marvellous! Thanks!

[@ko-Daegu]

this is the most complicated tut on the internet about this.. thanks

[@VivekYadav-ds8oz]

5:18 What was the point of having so many classes and factories and functions if the actual functionality is just shoved into one function?

[@isitanemail]

I was waiting for something that could explain in depth the full vulnerability,log4j and jndi. These videos helped me a lot in understanding. I would request to release such content more and probably a little quicker to help clear these things. Some other explanation caused a lot of confusion

[@deviousmethod1310]

thank you for another amazing work!

[@danbondarenko7894]

Thank you for the great explanation! I especially liked that you went step-by-step through the vulnerable code -- this helps me grasp what it might feel like to write vulnerable code. If you're looking for ideas of what to cover next, then I'd propose explaining what happens within JNDI lookups: when does remote code get executed? what would be a legitimate use case of this remote-code execution part of JNDI? Thanks again.

[@Dropje86]

Man, you have no idea how many arguments I’ve had with developers about levels of abstraction. The DRY principle has been taken way out of its original intent and people now just add layers of indirection for the sake of it. So often things can be kept much simpler by repeating yourself.

[@tambling3961]

Holy hell this library is bloated like there's no tomorrow 🤯 Besides the security issues I imagine that it must slow down your code by a significant amount... dafuq

[@john33john33]

one thing i learnt from many security issues. Try NOT to write your own parser famous browsers, frameworks screwed up and i ain't outsmart those engineers

[@desmond-hawkins]

Regarding the question at 7:14 about using ${env:HOME} vs log("... {}", System.getenv("HOME")): there's more to it than this trivial use case. While the example given here is of course less risky and still just as easy to read, expressions are also expanded from other places where you're really writing config, not code. For example, when logging structured messages (often to Splunk or similar) you often want not just the log message but also the hostname, the process name, the environment, maybe the cluster and namespace if you use Kubernetes, etc. You can configure Log4j to include such values as named fields in your structured log events, with config like . You're not writing code here, but config: this goes in log4j2.xml. What's the alternative if we block all such expansions? You'd likely have to write your own Layout class and implement each field in code. You could also have your own (non-expanding) config file to map environment variables to event fields, but then (1) now it's a lot more code to write and (2) it's still not as flexible as Log4j since you can't do things like having a field named "cluster-namespace" with the value "${env:K8S_CLUSTER}-${env:K8S_NAMESPACE}". So yes, expression evaluation *is* useful in the sense that you can have advanced logging configuration without code, which covers a vast majority of use cases. Adding any kind of network call to this system is insane, but it is certainly possible for this feature to be flexible and useful without going overboard.

[@TheNovakon]

Why every somebody needs lookups in realtime? We are processing logs on different node - all lookups can stay unresolved and eventually resolved later during processing - because realtime processing slows down a lot

[@desmond-hawkins]

@@TheNovakon Being able to query logs in Splunk or Logstash a few seconds after an event happens is *essential,* and working on any system without this ability is much more difficult. What do you do if you have to wait a while for the batch logs to be delivered? What happens during an outage if you don't have access to logs to understand what's going on?

[@siimk13]

Not to mention that any code change requires a roll out/deploy to production environment whereas with log4j you could just change the log config file and have log4j reload it. Easy as pie.

[@desmond-hawkins]

@@siimk13 Good point, and changing code also means extra build time, in almost all cases. It's certainly needed for Java since you have to compile it, but these days even interpreted languages have a "build" time if only to create a docker image or any other kind of package.

[@TheNovakon]

Any example please!!!

[@gubgubbubbub]

Awesome, informative video; thank you!

[@Klatuu82]

and another very good video. thank you for your work.

[@novelhawk]

4:50 Isn't it pretty much unoptimizable? How are you going to find a substring without iterating the string?

[@lassipulkkinen273]

I was wondering the same thing, although from when I was last using Java (years ago) I think there was an Iterator thing that was supposed to have a small performance advantage over c-style for loops by eliminating bounds checking or something... Also, maybe String.indexOf would be even faster if it has an optimized (i.e. non-JIT-ed) native implementation.

[@krzysztofdziembaa4982]

There are string searching algorithms, which may be faster than "brute force", but (according to an email in Newlib's mailing list) for short strings (which are quite popular) the naive algorithm is the fastest. Other algorithms (may) require preprocessing and/or memory to allow faster searches and whether this pays off depends on the processed data. Two-Way algorithm is one of such algorithms and is used in e.g. glibc.

[@lassipulkkinen273]

@@krzysztofdziembaa4982 I knew about those, but I don't think they are useful if the searched-for string is one character. While "${" is technically longer than that, the code *is* optimized to only look for "{" after it's found a "$", and "$" is not a character that's going to appear often in the input, apart from lookups of course. That said, the parsing could just be moved to an initialization stage if the "lookups in format parameter values" misfeature was removed, so... (maybe this is already being done; I don't actually know.)

[@ThePC007]

You do it using a finite state machine. Granted, the substring is so short that it likely doesn't matter much anyway.

[@cescfabregas4377]

Great video!

[@danthe1st]

+1 for mentioning insecure deserialization of untrusted data.

[@TimLF]

I agree log4j is pointless bloat. Spring (86% market share) and Tomcat (61% market share) are not using log4j, netty is using log4j.

[@BatteryProductions]

effin awesome man!

[@FUTUREPES]

Explanatory 💝

[@icenberg5908]

Amazing. Thank you.

[@berndeckenfels]

I remember the time when I was reviving the design concepts DJB used in qmail. One was a very careful logging concept. It was going in depth into timestamps (TAI logging has no problems with timezones and leap times) as well es escaping user provided data. It is just important that log messages can’t escape the log format (line feed and separator evasion). Once you are that thorough with design you would not 8ntroduce such lookups and expansion patterns, I always missed such a escaping system and structured handling of log message arguments in the typical java frameworks.

[@berndeckenfels]

7:35 I agree that env: lookup for the sake of easier logging is not a good security design, but I think the idea was using lookup in patterns or config declaratively. Looking up messages is just stupid,

[@ltxr9973]

Very good comment about hidden complexity. It's rampant in Java.

[@Ms.Robot.]

This is nice. ❤️

[@cybersecurity3523]

Good job bro 👍👍👍

[@lijnk]

The main reason the library has all this indirection and events is so it's thread-safe, easy to write new appenders (not everything uses just files and stdout), and so it's unit testable (good dev practices mean the code should be tested). If they didn't have that, then all the log messages would be jumbled up in the same line, it would be impossible to add new appenders like logstash or slack, and whenever a new release is made, the maintainers wouldn't be able to guarantee the project functionally works. With that being said, some eyebrows should have been raised when lookups were performed in the log message itself.

[@omri9325]

You don't need to defend the bad choices that were made there, the code is still shit.

[@ZintomV1]

@@omri9325 lmao

[@lijnk]

@@omri9325 The overall structure of any decent logging library is to take log events and pass them to appenders that write them somewhere, which the library does. Of course there's bad code within it, the noLookups section proves this, but saying the entire thing is garbage is about as asinine as saying security is only there to hinder developers. If you believe the whole thing is shit, you're welcome to contribute cleanups as it's open source.

[@omri9325]

​@@lijnk "saying the entire thing is garbage is about as asinine as saying security is only there to hinder developers" there is no logical progression towards that statement. I'm not going to to touch any of this shitty code, I'll rather contribute to projects that are healthier and have a future. Let that shit die.

[@GhostBitchKiller]

Ah Teil 2, sehr gut👍

[@BenjaminAster]

Und ich dachte immer, ich sei der einzige, der deutschsprachigen KZbinrn auch deutsche Kommentare hinterlässt.

[@GhostBitchKiller]

@@BenjaminAster 😀

[@AniltonNeto]

G.O.A.T very well explained :) LOF

[@tlouik]

respect to everyone who watches this on christmas :O

[@LeandroCoutinho]

"I'm a prophet now!" kkk Great job!!!

[@protiumdev]

Great video! I don't understand how companies do not audit their dependencies before adding them to their stacks. I hope it's also a lesson for the big companies that make use of open source libraries to start funding open source projects that add value to their products/business.

[@blizzy78]

That's very easy to answer: It takes a lot of time - and therefore money - to audit something. Especially when you're not intimately familiar with the inner workings. That's time and money companies are not willing to spend because it doesn't directly produce earnings.

[@jhbonarius]

Blizzy1978 is completely on the point. Imagine you're a small software development company. In order to get a contract, you have a small margin. You have to deliver within a few months. There's no time to develop every part and framework yourself, let alone test and mature it. Also: doing everything yourself is risky, because you often don't have enough experience with everything. I.e. we don't reinvent the wheel and use libraries everybody is using: if everybody's using them, they're probably mature is the assumption. There's no time and money to audit anything usually. Money doesn't grow on trees and it's a competitive market.

[@supermonkeyqwerty]

7:25 As a CTF'er by hobby and a developer by day job, I don't think this is a good idea from a library developer perspective in the first place. Whenever you provide 2+ different ways of doing the same thing, you're making code less consistent and harder to refactor in the future. Imagine that at some point in the future, you or your team decides to move off of environment variables and want to use a configuration file instead. You might think of searching for all usages of `System.getenv` in the project and replacing them, but that would leave this `${env:...}` logger config thing incorrectly untouched. Of course, it's only a cost, and that might be outweighed by a payoff. But I don't think that's the case for Log4j's feature, since the added convenience is tiny, developers might already be used to doing it the 2nd way as the standard, and it doesn't generalize. (What if you wanted to use the same string message in the logger and someplace else?) This happens much too often, because it's so easy to lose the forest for the trees. There've been times when I finally stepped back, and then realize that something was completely unnecessary.

[@lal12]

I think people are kind of looking at this issue the wrong way. Yes I agree the log4j is bloated and jndi is a terrible idea. However at the end of the day this is just a classic example of injection, where the user input was passed into the format string instead of as an argument (to use printf terms here).

[@supermonkeyqwerty]

@@lal12 Honestly, I haven't look at Log4j too closely myself, but does it even have an analogue to printf's `%s`? From its recursive substitute shown off here, I'm not convinced that such an option wouldn't just have the same vulnerability as using the string directly. Either way, whether understanding the entirety of a library's API and possible configuration is the _responsibility_ of a developer or not, it's not something that currently happens or will change anytime soon. I've got no clue what we'll do to prevent something like this in the future.

[@obsidianskin9502]

good stuff

[@rydmerlin]

Can you elaborate on how you get to remote execution though?

[@stanleycoffey]

maybe go watch the black hat talk

[@jasontiscione1741]

It happens through object deserialization which uses the remote class loader. The log4j programmers did not actually intend for object deserialization to take place, but it can be triggered. The log4j code creates an InitialContext and passes the JNDI name to lookup(), expecting a value that it will immediately cast to String to be logged. But inside lookup(), Java notices the crafted array with the serialized Evil object stream in the attacker's JNDI response and decides it can return an Evil object to log4j instead of a String, so it loads the Evil class and then proceeds to run the Evil static initializer. This crap was designed for the 90s Internet when Java meant applets.

[@tehwinsam3522]

Just want to wish everyone merry christmas !

[@TheBiggreenpig]

7:35 Yeah, when I learned about the JNDI and environment variable possibilities of log4j, I was wondering why not simply let the programmer retrieve those and pass them as a string. Log4j definitely does more than it should. In no code we ever used any lookup, but we had them enabled unknowingly.

[@jamess1787]

1) Design for simplicity. 2) get it working 3) revisit project 6 months later realizing you forgot target #1. *Ends up with 1000's of lines for some basic function*

[@timelessnesses]

*f a c t*

[@TheNovakon]

13:51 - THIS IS THE CORE OF THE PROBLEM! The whole format vulnerability is easy to understand but this is treachery!

[@codewithprabesh010]

Good Explanation. It's really helpful👍👌

[@alessandro.rossini]

Freaking 2013 maan! It's wild!

[@amyshaw893]

this thing about using less complex stuff for security reminds me of how i approach coding games. Recently I was working on something in a team, and I was like "ok, we need to play some sounds. we can use a simple sound manager and shove it through unity's inbuild sound system, done." But then this designer comes over and wants to use this huge complicated library that allows you to do mixing on the fly and spatial sound and allows the designer to make complex soundscapes without having to do any code and all this crap and I'm just like "...or we could just play some sounds, and you use audacity to make the sounds, and our git repo doesnt explode from all the new files, increasing clone and pull times"

[@firetf]

While i agree with the secure design philosophy, one big reason to have a logger internal formatter is performance. You often have tons of log calls to log levels that are not printed, and then the whole lookup and string formatting can be omitted, while if you do it manually, you would first look up all the things and build the formatted string, just to throw away all that work if it's a log call to debug level but you're on error level.

[@olafbuitelaar]

I Also think there is another aspect in this. Which is configurability. Like the environment variable case. It might not be included in the source. But in some production instance its useful to log this information, it can simply be added to the config. Instead of adjusting the code.

[@ruaidhridevery1458]

Devs get so much convenience and utility from _eval everywhere_ that preaching the simplicity of being more static is a really hard sell unfortunately. For this style of all-things-to-everyone library, it is nearly impossible for the devs not to embed interpreters.

[@TheMAZZTer]

This huge bug is caused by a desire to have string interpolation in Java, which doesn't have it. The problem is string interpolation is done by the compiler, meaning runtime strings (especially user input) can't trigger interpolation. String.Format looks like Java coders' best bet, but again, keep the format string static! But you won't have issues with recursive format specifiers (I would hope)! Example of string interpolation in other languages: JavaScript: `This is a string ${expression + goes + here}` C#: $"This is a string {expression + goes + here}" Looks very similar to how Log4j does it, but in both cases having ${} or {} in any of the variables won't trigger a string interpolation, so it's safe.

[@jasontiscione1741]

The bug was 10% caused by string interpolation and 90% by Java's eagerness to whip out a remote class loader and start executing mysterious crap without being asked, which seemed like a good idea in the 90s.

[@passerby4507]

@@jasontiscione1741 Funny thing, python pickle is the exact same.

[@kawcco]

@@passerby4507 I rarely see `pickle` in real-world code at all, and if it is it's usually for efficiently storing instances of Python's top-level data-types (strings, numbers, sequences, dicts) locally. Sending Java objects over the wire seems to be scarily common, however. I believe this is because Java's standard library lacks a convenient way to send data over the wire without touching XML. Us Pythonistas have `json` and a much easier to use XML lib, while Java users have *multiple*, painful XML libraries that must all be used together and no way to parse JSON without bringing yet another external dependency into the fold.

[@LiamDennehy]

@7:39 "Complexity is the enemy of security"

[@shemmo]

8:40 thumb up for this

[@SimGunther]

12:20 Proof that he's a time traveler 😎

[@altairbueno5637]

“It took me soo long to learn how to use maven with dependencies” Literally every java developer ever

[@ekfliu]

at least it's not gradle.

[@altairbueno5637]

@@ekfliu its Definetly not Cargo

[@greob]

Interesting! Made me thing hard about my own code for a moment. >_>

[@marcoantonio7648]

too much time to run Maven oh yeah, I felt that

[@bluejimmy168]

At 13:15, he said keep format string static to fix the problem. What does he mean by that? If a string is static then we dont need user input? If we need user input then it is not static? If I ask a user for input and want to output it back to the user then the string not static anymore? Thanks.

[@brunoais]

9:17: It may seem weird but I've been screaming about this kind of use of log4j since 2017, I think. However, I haven't seen any of your stuff until 2019. I didn't know log4j had jndi, btw. I stopped using it back in 2016/7, IIRC 12:51: This vulnerability with databases is very very prevalent! I never had such issue If I knew log4j had jndi, I would have known how dangerous the lib is much sooner. Thankfully, someone found it. Still, I wonder how many systems were attacked due to that.

[@jhbonarius]

Hahaha, all these endless (and typical Java) abstraction layers... and then you finally come to the parser, and that's just a much too long method written in a C-style way.

[@bibi2900]

7:40 this is actually how you SHOULD write software. From a software engineering perspective. This is the KISS (keep it simple stupid) principle. This library is over complicated for no reason… But take this aside… you should never parse strings to code.. same as you shouldn’t use exec/eval etc.

[@lal12]

I mean it depends how you mean it, but quite often it is necessary.

[@mrkretschmer]

what if I replace the lookup function directly with return null ? =) fixed ?

[@passerby4507]

Classifying log4j string substitution as syntax sugar isn't exactly fair. It allows for recursive lookups, e.g. have an environment variable store what's supposed to be looked up. Now that may or may not be what you want, but it's not the same thing.

[@TheNovakon]

Please any use-case to have this; ${env:WHATEVER} and put into env variable WHATEVER=${jndi:ldap:...}. ??? Why I need this? However even content of enviroment variable should not be trusted...

[@rastabong420]

@14:59 am I watching an old Godzilla film? 😁

[@ZintomV1]

2:19 - the reason why most Java applications are way slower than their C++ equivalent (I'd argue even C# equivalent)... Not because of the JIT, but because of needless object creation and abstraction, if you just need a simple string, leave it as such! No need to wrap a class around it!

[@omri9325]

My theory about that is that most of the java ecosystem was created at a time were design patterns where THE THING, and everyone jumped in adding patterns without even knowing when to add them, people were forced to add them just to not look idiots in front of their peers. When a peer adds something with many patterns/abstractions. you don't doubt that it's too much abstraction, it might be that you are "too stupid" to understand and you let it slide. Languages that evolved in modern times don't include all those AbstractFactoryManagerImpl shit, they add abstractions only when they are needed.

[@-morrow]

object creation is very cheap in java due to it's memory allocation strategies and is mostly faster than C++ new object creation. besides, for almost all java applications the kinds of speed improvements you get from using C++ instead of java or optimising object creation are dwarfed by the actual bottlenecks (round-trips, db queries, disk seeks, time complex algorithms, etc.)

[@jhbonarius]

@@-morrow you're comparing apples with oranges. Sure maybe the default "object creation" of Java is faster, but C++ is not an object oriented language like Java is. You have much more flexibility. And if you write a custom allocator and pre-allocate, C++ will likely be faster. But: you have to do everything yourself. It's a lower level language.

[@jhbonarius]

Their equivalent? What applications are you talking about? C++ and Java play different roles in the programming ecosystem. They each have their strengths and weaknesses. You wouldn't expect applications to be written twice in each of them, as there's no purpose to that. Instead you would expect larger applications to have parts written in either and some interop between them.

[@omri9325]

​@@-morrow Bottlenecks like the huge amount of extra memory needed by JVM applications. The classloading system in java is painfully slow on program start-times. The only thing java can do faster than C++ is on optimizing things it learns to predict from the runtime. C/C++ can technically optimize everything better, but with a lot more effort to hand craft them.

[@krzysztoflewandowski8262]

CVE-2017-5645 was kinda first on the same

[@mskiptr]

Tbh, format strings really are an anti-pattern. They conflate two different functionalities and make it less obvious what should happen where. They can be made safe (vide printf in Idris) but in such case it's an absolutely pointless complexity. We should just stick to clear and unambiguous abstractions. If we want to log some string literal, we should be able to just do so right away (without any need for escaping or lookups). If we want to log a value of some variable it should work the exact same way. And if we want logs to be parametrized with the future environment, then we should either use lambdas (Environment -> String), or some specialized data structure and not a string-but-with-an-overloaded-meaning!

[@Ostap1974]

@13:45 -- was my first thinking as well... Why would anyone ever in any language write a text fromatiing function that has a formatting handler outside of the format string.

[@voidedname]

I'm a software engineer, and I hate most magic libraries. ORMs are my biggest gripes, log4j is another, spring boot is a huge issue... Magic is always bad, it exponentially increases the mental complexity, all the magic that isn't clear from the code has to be kept in mind. But sadly, in the java world at least, you're more or less forced into using them. There are ways to make the code prettier, I won't get into it. But I absolutely hate invisible side effects. If I want my code to cause side effects then I damn better actually tell it to do that side effect. The reason it's so popular is because it (wrongly) seems like you can write code faster and more efficiently. That is true when prototyping and in the initial phase of a project, but the hidden complexity quickly swallows it after just weeks where you spend an ungodly amount of time and mental effort to maintain and develop further features...

[@ZucchiZ]

Hey security researchers, you missed the *"rediculous"* example

[@roboknight]

Unfortunately, you won't be able to use an XML file, as Log4j does, to enable configurable logging if they just use in-line environment variable look-ups. However, the real question is, do they really need this kind of flexibility in a logger? Maybe or maybe not. Maybe they could have parsed the file way before the library goes and does anything with it to help secure it. But it is difficult to get simpler when you want flexibility. Hard coding does help secure some things, but can also introduce other bugs (buffer over-flows come to mind), simpler or not.

[@amarmohmed1648]

I have the same passion that you had when I was young, and more than that, I saw THE last video and I did not understand anything, so I saw your first video, which is the intro video Please, would you teach me the name of that passion?

[@C10udburst]

wouldn't it make more sense to make the lookups into a different library for example? why is this a part of a logger??

[@code913.]

It's not about the code being a ridiculous example. It's actually about drive and power

[@Torterra_ghahhyhiHd]

8:37 i think developer like to introduce this kind of complexity for 2 main reasons 1 he want to be needed in the job he is hoping to add new capabilities more power to the program the program like the art of the artist have some subjectivity and is never finish always want to ad some extra features or hidden potential that only the creator can exploit more.

[@fabricio5p]

Honestly this code feels like they intentionally wanted to go fancy with the architecture and end up with a classic freaking mess

[@drakey6617]

10:47 in my opinion is not always correct. They first arguments can actually also be in registers and not on stack. Stack is used when the first registers are full. At least this is for ARM

[@dzaima]

"here's how it looks like with a simplified stack in memory"; 32-bit x86 passes everything on the stack, for example

[@drakey6617]

@@dzaima Yeah, true. I suspected that this is true for x86; that is why I said "not always" and "for ARM"

[@mapp0v0]

You have reminded me, as someone who hasn't programmed in years why I feared java. Of course this bug was discovered earlier. Didn't you see the reaction on the Chinese. I am sure the Five-eyes are equally pissed off.

[@dummypg6129]

When people make feature-rich apps ending up with tons of vulnerabilitu.

[@justknot4481]

this is 🤫😂

[@thesenuts4472]

As a hacker sorry developers no love it is what it is.

[@jasontiscione1741]

Technically this shouldn't be a problem because you're not supposed to pass untrusted user input to the logger in the first place. They might as well put warning labels on Q-Tip boxes saying "Don't stick these in your ears".

[@cdhagen]

Well, but to sanatize user input, you have to know what you are actually looking for. If you don't know about the existence of this crazy lookup feature, how would you be aware of the problematic character sequences?

[@jasontiscione1741]

@@cdhagen This is what happens when you give data too much control over what the CPU does, as opposed to the compiled source.

[@jhbonarius]

@@cdhagen although many people do that, it's difficult to comprehend what all "user input" is. I.e. in the case of log4j, I saw somebody change his hostname to a string with the jndi vulnerability. Thus, when the logger would print the hostname, it would interpolate that string, executing the embedded code. Would you have tought of considering a hostname "user input"?

[@dynamicfield]

well there's actually a warning to not put Q-Tips on your ears, it can be dangerous, lol

[@sikkavilla3996]

linus, thank you for providing an in depth analysis to this vulnerability. Very much appreciated!

[@Yaxqb]

This honestly made me fear Java more than Log4shell: inconsistent if-else statements, 1000 layers deep call stack. So much fluff!!! Java attracts people with bad taste. Architectural astronauts. Python for life

[@FalcoGer]

The principle of single responsibility. A logging library is supposed to log messages. It shouldn't format your string. If you need to format your string, then add a class for that or use the basic language features.

[@ZintomV1]

4:50 - May be part of the cause of why Minecraft runs so slow, I imagine there's a significant performance uptick if a server enables noLookup and this whole "loop over each char" thing is skipped.

[@wouter11234]

Looping over each character of a string causes nearly zero performance overhead, unless you are getting tens of thousands of logged characters per second, in which case you'd be having other problems

[@stewartzayat7526]

Using java causes massive performance overhead.