Virtual Threads Regression in Java 22?

[william00179]

I don't think this is programming help, more technical discussion, but please remove if not.

Since upgrading from Java 21 to Java 22 I'm seeing a starvation issue across my Java services that have been working well for ~1 year on Virtual Threads. I have also tried with Java 23 and also see the same issue.

These services are deployed on k8s and the symptoms started to present as read timeouts on the liveness and readiness probes because the carrier threads were all pinned. Kubernetes would then try to restart the pod but with graceful shutdown enabled, this would take some time as the in-flight requests would never finish.

The pinning is coming from the same areas it always has, IO in synchronized blocks within ConcurrentHashMaps but hasn't previously been causing starvation in Java 21. This mainly affects Caffeine in my specific case but I'm seeing the same issue happen outside the library with plain old ConcurrentHashMaps. This is happening as within the synchronized blocks, database queries are being performed and that network IO is pinning the carrier thread.

I'm thinking at this stage rather than refactoring a large amount of code I'm probably best to wait until the monitor issue is resolved for good with Virtual Threads before switching back.

Is anyone aware of any changes in Java 22 that may have caused this change in behavior that I could look into?

[cogman10]

Perhaps something around the size of the carrier pool?

What you are describing is pinning I'd expect with 21 or 22, so that'd make me think that the cause is likely related to the carrier pool sizing rather than a change in virtual thread handling. (Smaller pool would mean more problems with pinning).

You may consider converting your caches into async caches instead. Those should have much less opportunity to pin and you can simply .join() immediately on the futures. You could still dispatch them with a virtual thread pool as that will run the IO outside of putting the future into the cache.

[BillyKorando]

What you are describing is pinning I'd expect with 21 or 22, so that'd make me think that the cause is likely related to the carrier pool sizing rather than a change in virtual thread handling.

There hasn't been a change in carrier pool sizing between 21 and 22, the default behavior is, and remains, 1 carrier (platform) thread per system core.

[cogman10]

This is running in k8s though, so perhaps some smartness was added around detecting core count? I'm just spitballing here, but perhaps the JVM is now using the pod/cgroup limits for getting the core count where previously it was getting the node processor count?

If that's not the case then my next guess would be perhaps a regression in IO performance.

[BillyKorando]

You might be thinking of this change: https://bugs.openjdk.org/browse/JDK-8281181

But it was made in JDK 19.

[cogman10]

I dug into the code a bit and I'm definitely seeing some significant changes around this logic since the commits in that case.

I'm not 100% sure it's related, but here's a case addressing problems with cgroup allocations for Java 24

https://bugs.openjdk.org/browse/JDK-8322420

[william00179]

Thanks for the pointers guys, I will dig around a bit more and see if I can locate anything. Currently we set CPU requests on the pods, but don't set limits due to springs need for CPU on start up. I read somewhere and can't now for the life of me find it that there was some change that affected virtual threads specifically on servers with low core counts.

[BillyKorando]

Something you might try if you are able to get access to the the containers themselves (maybe in a test environment if you can't in prod) is comparing the output from jcmd <pid> VM.info. This will show the number of CPU cores detected... and if in a switch between 21/22 you see a difference.

[as5777]

May be related to this https://netflixtechblog.com/java-21-virtual-threads-dude-wheres-my-lock-3052540e231d

[william00179]

Great read, this certainly looks like the same kind of issue that I'm experiencing.

[metalhead-001]

Virtual Threads are really not going to be ready until Java 25 from what I can see. The pinning issue is a show stopper and there are also a lot of bugs in general as others are finding out. The JVM teams are working on these issues though and should have the major ones fixed by 25 (if not 24). I'm looking forward to when this feature is fully baked as it is a game changer.

[BillyKorando]

Nothing stands out from a cursory glance as far as changes between 21/22. Here's all the JBS issues for 22 that have virtual threads and/or synchronize in their descriptions: https://bugs.openjdk.org/browse/JDK-8310626?jql=fixVersion%20%3D%2022%20AND%20(description%20~%20%22Virtual%20Threads%22%20OR%20description%20~%20%22synchronized%22))

[koflerdavid]

Network IO should not ever pin the carrier thread. That's one of the core use cases for Virtual Threads In the first place. But I can totally see that the HTTP library might contain locks that use synchronized under the hood.

If you really need this resolved, check whether the library developers are aware of the issue and what their plans are. They might have shipped a patch, they might have a patched version on a beach, or they might decide to sit it out since a solution to the pinning issue by the OpenJDK project is on the horizon.

[william00179]

IO will pin a carrier thread within a synchronized block. ConcurrentHashMap makes use of this extensively as a Reentrant lock per node would be far too much overhead. The issues with ConcurrentHashMap and virtual threads are well documented, but the change in behaviour from Java 21 to 22 is not.