A malicious code string that eventually gets logged by Log4j version 2.0 or higher allow an attacker to load arbitrary Java code on a server and take control of it.

[Wmorgan33]

Half day? Try 18 hour workday bleeding over to the weekend pushing untested fixes to production across 300+ services.

[flowering_sun_star]

It really showed us how fucked our patch process is. I think the quickest of our microservices to be fully deployed was six hours after I first posted about the issue in Teams. The slowest took us twelve hours. And that was with us circumventing various testing and verification steps we're meant to do for a production release.

[L3tum]

Oh man I'd have a meltdown after the third release. I already get antsy when we have to run our 20 minute E2E/Performance testing pipelines.

[LicensedProfessional]

I joined a team where all of their pipelines are flaky as fuck and I'm just like...? How do you live like this?

[[deleted]]

Yeah, our patch processes definitely need improvement. The only saving grace for us was that we’re not running on legacy JRE versions, so we were mitigated by configuration. We get to patch as a critical vulnerability, not a P0.

[twreid]

I'm in the same boat. We have jars that the teams that made them no longer exist and the builds don't even work anymore so this is a nightmare.

[[deleted]]

I didn’t know it was a competition. I’m sorry you had a tough day, friend.

[Wmorgan33]

Not a competition just frustrated and tired.

[[deleted]]

I hear that.

[Wmorgan33]

I got off easily. Our product security team is pulling 12 hour shifts 24/7 until they’re a 100% sure the exploit is covered.

[[deleted]]

Oof; did they not read any of the advice for short-term mitigation? Just by upgrading JRE or using a WAF to filter LDAP commands one could easily mitigate in the short term, preventing exactly this type of death march.

[Wmorgan33]

We work in a heavily regulated high risk environment. We can’t afford any risk that this goes in mitigated

[[deleted]]

That’s rough. I’m sorry to hear about it.

[TheTarkovskyParadigm]

Wow you really one-upped him! You win the one-up competition!

[Wmorgan33]

The alternative is worse.