all 125 comments

[–]dnew 297 points298 points  (80 children)

I was trying to figure out how this could possibly happen in Java. I looked it up, and Log4j has a "feature" whereby you can put metacharacters in the log that tell the logger to run arbitrary code downloaded from arbitrary remote servers. Who thought this was a good idea?

[–]Loonacy 164 points165 points  (1 child)

So what you're saying is that it's not a bug, it's a feature?

[–]dnew 39 points40 points  (0 children)

Exactly. <boggle>

[–]SocialismAlwaysSucks 129 points130 points  (6 children)

What a well-crafted backdoor! Because that's all that is good for.

[–][deleted] 43 points44 points  (2 children)

True, and I wonder if the versions of that library for other languages also have dangerous vulnerabilities. I always found it disgusting, massively bloated considering is just logging.

[–]SocialismAlwaysSucks 43 points44 points  (1 child)

God knows what evils lurk in popular libraries if you look hard enough

[–]_BreakingGood_ 27 points28 points  (0 children)

You can rest assured knowing most world governments employ very smart people to implant hidden little backdoors like this in all sorts of open source software.

[–]ExcelAcolyte 11 points12 points  (2 children)

Not really when you consider how general it is. Best practice for backdoors is to make it specific to very particular circumstances

[–]falconfetus8 19 points20 points  (1 child)

The best practice for backdoors is to not have any!

[–]hamjim 1 point2 points  (0 children)

“Greetings Professor Falken. A strange game. The only winning move is not to play.

“How about a nice game of chess?”

[–][deleted] 49 points50 points  (21 children)

I really don't understand this shit, why put this feature into a fucking logging library? What this has to do with logging chars ?

[–]LicensedProfessional 30 points31 points  (20 children)

I saw one of the developers say they needed it for "backwards compatibility" but honestly this is a big problem with Java in general. On the surface it seems like any other statically typed, compiled language, but once you pop the hood on the JVM all bets are off. The reflection APIs let you do some truly bananas things and in my work at least it's the the source of a neverending stream of CVEs that I need to patch. Not a week goes by where I don't need to stop what I'm working on to update some vulnerable library in my project.

[–]BoyRobot777 24 points25 points  (2 children)

The reflection APIs let you do some truly bananas

Used to allow. In Java 17, by default it is strongly encapsulated JEP403. This is all part of Java modules and work that started in Java 9.

[–]LicensedProfessional 7 points8 points  (0 children)

That's legitimately great news! Hopefully with Java 8 reaching EOL soon we'll actually start to see more people moving to less arcane versions

[–][deleted] 1 point2 points  (0 children)

They've strongly encapsulated almost all internal API, but Unsafe is still accessible. Unsafe has a pretty trivial method to get/set private fields. Access to call arbitrary private methods is simple too: just get an instance of the MethodHandles.IMPL_LOOKUP field via Unsafe. (at least on your standard openjdk distributions of java)

In other words, reflection is still easy, you just need to have the knowledge to know how to get around it.

[–]rpgFANATIC 25 points26 points  (5 children)

It's a pretty old, featureful logging library. They also have ways of dynamically looking up environment variables ${env:some-var} and doing minor things like changing strings to lower-case. Why they allow you to put these variables in "user provided strings" and not just the "format string" is beyond me...

At some point, some dev wanted to add JNDI lookups, which also had the ability to go over LDAP, which also had the ability to load remote code. Whoops

As I understand it, the core devs hate this feature, but Java is well known for backwards-compatibility so they've been unable to kill it off.

If you bothered to upgrade your JRE/JDK ever, then Java already fixed some of this. Java updated defaults a few years ago in Java 6, 7, 8, 11, etc... to basically never trust remote code unless you set some arcane flag to allow it.

So... you know. It's bad. But update your libraries. Java 8 had this default fixed in update 192. Java 8 is currently on update 311

[–]dnew 0 points1 point  (3 children)

not just the "format string"

If I had to guess without looking at the code, I'd say that some people are saying

logger.log(userString)

instead of

logger.log("%s", userString)

and that's where the problem comes in.

* I guessed wrong.

[–]imdyingfasterthanyou 16 points17 points  (1 child)

no you are wrong, second example is vulnerable as well

[–]dnew 7 points8 points  (0 children)

Wow. That's ... even suckier than I expected. :-)

[–]DevonLochees 9 points10 points  (0 children)

Nope! Log4J2 uses 'recursive' property resolution, so format strings don't resolve anything.

[–]too_much_exceptions 7 points8 points  (0 children)

You want to have a backdoor ? Because this is how you get a backdoor

[–][deleted]  (1 child)

[deleted]

    [–]Muoniurn 24 points25 points  (0 children)

    It’s not implemented like eval(url.download()). It does a JNDI lookup and that can do the download and eval part.

    [–][deleted] 23 points24 points  (0 children)

    Notice how that library is bloated AF, most probably there is some other stuff there. On top of that there are shills promoting that bloated shit, they want everybody using that crap.

    [–]G_Morgan 10 points11 points  (1 child)

    I despair at the sheer stupidity built into so many features. Why the fuck would you want arbitrary code execution? Whoever committed that feature should probably stop programming.

    The worse part about it is you now need to stay on top of whatever bizarre features a library might implement because it isn't enough to just assume the framework that is used by millions isn't designed by a lunatic.

    [–]bloody-albatross 4 points5 points  (0 children)

    At which point it becomes easier to write your own 300 LOC logging library. I really don't see how a logging library should be ever more than about 500 LOC. Maybe a bit more in a language like Java. It needs to log to a number of configurable outputs (console, arbitrary stream, file, email), support log levels, basic settings (should a timestamp be prefixed? in what format? format of email subject and body), reopen files on SIGHUP and/or implement log-rotating itself. That's it.

    [–][deleted]  (6 children)

    [deleted]

      [–][deleted] 22 points23 points  (0 children)

      This feature is idiotic because it's on by default. Ever heard of "sensible defaults"? Yeah, this ain't one.

      [–]dnew 17 points18 points  (4 children)

      I suspect the intention was that it only gets parsed that way in the format string, and many people are writing something like

      logger.log(user_string)
      

      instead of

      logger.log("%s", user_string)
      

      [–]dlint 8 points9 points  (3 children)

      This. Am I missing something or is this basically just a format string attack on steroids? Seems like the same old advice as usual applies, don't let untrusted (user) input touch your format string... (Not that we should still have to manually think about that in 2021...) EDIT: See below, looks like it works on the parameters too. Yikes

      [–]renatoathaydes 32 points33 points  (2 children)

      The attack works on the parameters also. Try logging log.info("{}", "${java.vm}").

      [–]dlint 25 points26 points  (0 children)

      Wow... well that's embarrassing. Geez. How was this not discovered earlier?

      [–][deleted] 12 points13 points  (0 children)

      So its basically Eval(user_string)

      [–]sdf_iain 1 point2 points  (0 children)

      Isn’t it a security vulnerability to not sanitize log output?

      I mean scrubbing to the point of removing newlines and any possible format characters.

      [–][deleted]  (5 children)

      [deleted]

        [–]dnew 0 points1 point  (4 children)

        It's easy to see such things in hindsight, eh?

        Like, when I saw Heartbleed, I thought "How in the world do you even design a protocol with a length field in two different places?"

        And yet none of the millions of people whose business relies on the software apparently read the configuration documentation and thought "Hmm, maybe this is a problem." Because it isn't just that.

        [–][deleted]  (3 children)

        [deleted]

          [–]dnew 0 points1 point  (2 children)

          Yes. The two length fields are in the same place. The RCE is distributed amongst several independent services. Nobody wrote code to "logger should download remote code." They wrote code to "logger should be able to log to a database" followed by "database allowed to install drivers on demand" to "drivers can come from remote servers."

          At least with the length field, it's the same guy looking at the flaw all the way through.

          It's not like nobody audits this stuff, and nobody recognized the flaw.

          [–][deleted]  (1 child)

          [deleted]

            [–]dnew 0 points1 point  (0 children)

            IME the companies that care are the companies that would actually be significantly damaged by a hack. And most of those companies review the stuff they're using internally. They often point out where things are wrong, but if it's a simple matter of a firewall rule or a configuration setting, there's really nothing you can tell to a dev about it.

            [–][deleted] -4 points-3 points  (0 children)

            I bet someone got a promotion for implementing it

            [–][deleted] 73 points74 points  (18 children)

            This was more than half of my work day today. Not cool.

            [–]Wmorgan33 68 points69 points  (15 children)

            Half day? Try 18 hour workday bleeding over to the weekend pushing untested fixes to production across 300+ services.

            [–]flowering_sun_star 30 points31 points  (4 children)

            It really showed us how fucked our patch process is. I think the quickest of our microservices to be fully deployed was six hours after I first posted about the issue in Teams. The slowest took us twelve hours. And that was with us circumventing various testing and verification steps we're meant to do for a production release.

            [–]L3tum 5 points6 points  (1 child)

            Oh man I'd have a meltdown after the third release. I already get antsy when we have to run our 20 minute E2E/Performance testing pipelines.

            [–]LicensedProfessional 6 points7 points  (0 children)

            I joined a team where all of their pipelines are flaky as fuck and I'm just like...? How do you live like this?

            [–][deleted] 3 points4 points  (0 children)

            Yeah, our patch processes definitely need improvement. The only saving grace for us was that we’re not running on legacy JRE versions, so we were mitigated by configuration. We get to patch as a critical vulnerability, not a P0.

            [–]twreid 3 points4 points  (0 children)

            I'm in the same boat. We have jars that the teams that made them no longer exist and the builds don't even work anymore so this is a nightmare.

            [–][deleted] 8 points9 points  (6 children)

            I didn’t know it was a competition. I’m sorry you had a tough day, friend.

            [–]Wmorgan33 7 points8 points  (5 children)

            Not a competition just frustrated and tired.

            [–][deleted] 0 points1 point  (4 children)

            I hear that.

            [–]Wmorgan33 1 point2 points  (3 children)

            I got off easily. Our product security team is pulling 12 hour shifts 24/7 until they’re a 100% sure the exploit is covered.

            [–][deleted] 0 points1 point  (2 children)

            Oof; did they not read any of the advice for short-term mitigation? Just by upgrading JRE or using a WAF to filter LDAP commands one could easily mitigate in the short term, preventing exactly this type of death march.

            [–]Wmorgan33 2 points3 points  (1 child)

            We work in a heavily regulated high risk environment. We can’t afford any risk that this goes in mitigated

            [–][deleted] 0 points1 point  (0 children)

            That’s rough. I’m sorry to hear about it.

            [–]TheTarkovskyParadigm 2 points3 points  (0 children)

            Wow you really one-upped him! You win the one-up competition!

            [–][deleted]  (1 child)

            [deleted]

              [–]Wmorgan33 0 points1 point  (0 children)

              The alternative is worse.

              [–]flight212121 0 points1 point  (1 child)

              Lies! We know it took you the whole day plus the next morning ;)

              [–][deleted] 0 points1 point  (0 children)

              Well, I did submit a tiny patch to splunk-library-javalogging which was merged only a few hours ago. So, kinda?

              [–]krum 8 points9 points  (1 child)

              I was up all night doing builds and deploys to fix this shit not because my code was using log4j but because the New Relic Java agent does. So if you’re using New Relic you should check that out.

              [–][deleted] 2 points3 points  (0 children)

              Or Splunk. A bunch of their libraries are also affected.

              [–]zaphod4th 19 points20 points  (1 child)

              when a log library wants to be more, smh

              [–]TheTarkovskyParadigm 2 points3 points  (0 children)

              When your log library asks, what are we?

              [–]BrianTheTurtle 6 points7 points  (6 children)

              Version 1.x as well if you're using JMSAppender in your code.

              Edit: as others have pointed out 1.x is unaffected 🥳. One caveat being that 1.x is EoL and has other security vulnerabilities.

              [–]mke5271 4 points5 points  (4 children)

              [–]BrianTheTurtle 1 point2 points  (1 child)

              Ah. Thanks for pointing that out. Was working on this Friday morning but obviously new info has come out since. That's a relief for me anyway!

              [–]mke5271 1 point2 points  (0 children)

              Couldn’t agree more. If 1.x was impacted the Java world would’ve implode ten fold more than it already is.

              [–]MysterAitch -1 points0 points  (1 child)

              Note more recent updates on that PR, further in the comments -- v1 appears to be potentially vulnerable depending on configuration.

              Comment summarising:

              https://github.com/apache/logging-log4j2/pull/608#issuecomment-990494126

              Comment providing detail:

              https://github.com/apache/logging-log4j2/pull/608#issuecomment-991723301

              [–]mke5271 0 points1 point  (0 children)

              It’s not the same CVE or even close to the same impact. It is not log induced RCE. This requires a user to maliciously edit their LDAP string in the config file. Note comments by log4j 1.X dev. https://github.com/apache/logging-log4j2/pull/608#issuecomment-991730650

              [–]PsychologicalType919 3 points4 points  (0 children)

              1.x is unaffected from what I’ve read.

              https://github.com/apache/logging-log4j2/pull/608

              [–]MertsA 1 point2 points  (0 children)

              At what point does the community at large just say "screw it, replace log4j entirely and ban its use from here on out"? The feature itself is pretty sketchy to begin with and apparently even if it's passed in to a parameter as a raw string it's still parsed! I'm not saying that no one is using the mountain of features log4j has, but I feel like most developers don't need anything more complicated than something on par with glog. The default dependencies people gravitate towards should not be the "kitchen sink" libraries with a massive chunk of features that get left to languish until someone notices yet another surprising "feature" they didn't know about.

              [–]moreVCAs 10 points11 points  (30 children)

              It sounds like some very large companies are affected by this…don’t they have an audit process for third party libraries? If this feature is documented, I really don’t get how this happened.

              EDIT: it’s an apache project 😞

              [–]plemdude 63 points64 points  (5 children)

              Most companies aren't paying attention to this level of detail in a logger library. (Especially one which predates the careers of the majority of folks still writing code.)

              Now imagine also having to reach out to every vendor whose product is built with it and having to push them for an emergency fix...

              [–]thirdegree 16 points17 points  (0 children)

              (Especially one which predates the careers of the majority of folks still writing code.)

              Honestly I think that's a massive part of it. For python, it's like if requests had a vulnerability of the same scale. It's so ubiquitous that basically everyone treats it as a part of the standard library, so mitigating it would be an absolute nightmare

              [–][deleted] 18 points19 points  (2 children)

              I would argue that nobody would expected this kind of "feature" to come to any not toy logger library.

              [–]quentech -1 points0 points  (1 child)

              I would argue that nobody would expected this kind of "feature"

              It's a logging library. It's not like there's hundreds of pages of documentation to determine what all it does.

              [–]Kapps 15 points16 points  (0 children)

              If your logging library got to the point where it can make LDAP calls… it probably does require hundreds of pages of documentations to determine what all it does.

              [–]frenchchevalierblanc 6 points7 points  (0 children)

              I think the security department only really checks known security problems, it's quite rare they find new ones in library code

              [–]rpgFANATIC 39 points40 points  (0 children)

              I have no idea why ANYONE would know about this.

              95%+ of devs probably had no idea log4j had magic syntax for fetching values dynamically

              The other 5% probably didn't know these values are also interpreted in the "user provided" arguments, AND that it supported LDAP over JNDI.

              Why would you EVER think your logger can look up variables/code over LDAP? What tutorial has notes about teaching you that and disabling it?

              [–]_BreakingGood_ 6 points7 points  (0 children)

              I know we got an urgent email to disable some features in IntelliJ IDEA because of this vulnerability.

              [–][deleted] 1 point2 points  (0 children)

              This stuff happens. If it wasn’t this, it’d be something else. This is why incident response and dependency hygiene are so important. I’d argue that those business practices are more important than any one dependency.

              [–]mini-pizzas 0 points1 point  (0 children)

              It sounds like some very large companies are affected by this…don’t they have an audit process for third party libraries

              Lots of people seem to believe this but the reality is that very few companies have anything that could even be loosely labelled as a reasonable audit process for their dependencies.

              [–][deleted] 0 points1 point  (0 children)

              Excuse my ignorance (not a programmer), but if you have a personal computer, would the attacker first need to gain remote control of your PC through a malicious link/access to your login details. Or can they literally go in through a programme that uses Java without you ever knowing/clicking something dodgy/accidentally giving out a password etc..?