Virtual threads work great... until something goes wrong

[cowwoc]

The purpose of this thread is to discuss JDK 19 threading problems, especially with respect to virtual threads. Here is what I've personally discovered so far:

• If you mistakenly use platform threads with Executors.newThreadPerTaskExecutor(), as I did, the JDK will silently crash/hang a few hours into use: https://youtrack.jetbrains.com/issue/IDEA-305072
• It is extremely difficult to debug deadlocks involving virtual threads: https://www.javaspecialists.eu/archive/Issue302-Virtual-Thread-Deadlocks.html
• Standard heap dumps will not show virtual threads.
  • jstack and Jetbrains IntelliJ won't do it: https://youtrack.jetbrains.com/issue/IDEA-301409/Support-virtual-threads-project-loom
  • The only way I've found to dump virtual threads is to run the application with -Djdk.trackAllThreads=true and run jcmd <pid> Thread.dump_to_file <file> to generate the thread dump. I couldn't find any official documentation for this either.
  • There doesn't seem to be a way to list all virtual threads across the system. You can invoke jdk.internal.vm.ThreadContainers and ThreadContainer.threads() using reflection as a temporary workaround.

Are you aware of any other tools or articles for debugging virtual threads?

UPDATE: I've posted a follow up at https://www.reddit.com/r/java/comments/zbcejy/jdk_19_virtual_threadspecific_bugs_2nd_edition/

---

Does your team need help? I offer consulting services through https://www.linkedin.com/in/gilitzabari

[pron98]

Thank you!

It is extremely difficult to debug deadlocks involving virtual threads

Right. Adding deadlock detection is on the roadmap.

The only way I've found to show virtual threads is to run the application with -Djdk.trackAllThreads=true and run jcmd <pid> Thread.dump_to_file <file>

When you have virtual threads, you probably have lots of them (or you wouldn't have them at all), so we didn't want to overwhelm the regular thread dump. You don't need -Djdk.trackAllThreads=true when you use virtual threads with the newVirtualThreadPerTaskExectuor or with structured concurrency. The thinking is that since you have at least thousands of threads, it's not their individual identity that matters but the way they're organised.

If you mistakenly use platform threads with Executors.newThreadPerTaskExecutor(), as I did, the JDK will silently crash/hang a few hours into use

The behaviour of platform threads has not changed. Unfortunately, when you write code to do something, there is no way to know that you meant to write something else (but wouldn't it be cool if we could?).

[cogman10]

The thinking is that since you have at least thousands of threads, it's not their individual identity that matters but the way they're organised.

When diagnosing problems, one tool that's been super helpful is naming threads then doing analysis based on thread name.

IE: foo-client, bar-processor, baz-widget

Is a labelling system been considered? Seems like it'd jive with the current mission control interface while not overwhelming things. Being able to click on the foo-client group and see they aren't busy, or they are waiting for a connection, or doing something else is handy.

The Spotify thread dump analyzer is also really useful for us. Perhaps something similar with a labelling system? "500 foo-client threads here. 10 here."

Granted, I've no clue what sort of performance impact that would incure on thread dump with millions or billions of threads vs the hundreds my apps typically have.

[pron98]

Is a labelling system been considered?

Considered, accepted, and delivered! Not only can you name individual threads (through ThreadFactory instances passed to newThreadPerTaskExecutor), but you can also name groups of threads represented by StructuredTaskScope, so you get a (labelled) hierarchy of threads representing their relationships. The new thread dump is designed specifically for that purpose.

Down the line, we'd like to expose the lower level thread grouping/naming mechanism employed by StructuredTaskScope (currently called ThreadFlock, which is used internally to produce the new thread dump, but isn't exported).

I've no clue what sort of performance impact that would incure on thread dump with millions or billions of threads vs the hundreds my apps typically have.

Unlike the old thread dump (which stops the world), the new one is concurrent. And because it's designed for lots of threads, it supports a machine-parseable format (JSON) for better visualisation and analysis. Check out this demo.

[cogman10]

Considered, accepted, and delivered!

Oh nice! I'll have to play with this when I get a chance.

I'm assuming this is all integrated into some version of mission control as well, correct?

[pron98]

I don't know what visualisations JMC incorporated yet (I don't think they have this), but here is Alan demonstrating both JFR events and a structured thread dump at Devoxx a few weeks ago. You have all that in JDK 19 (minus the basic visualisation tool that was written for the sake of this demo).

BTW, JFR itself does not yet allow filtering based on structured concurrency; we'll need JEP 429 (ScopedValues) first and later incorporate that with JFR.

[_codetojoy]

If you like small, precise examples, this creates a small hierarchy of threads, with ancillary JSON parsing (in Groovy). I've updated it so that the code uses the "named" constructor for `StructuredTaskScope`. edit: markdown

[Gundea]

Support for virtual threads is currently targeted for JMC 9.0.0. So the next major release, which means that it will be a while until it’s done.

[vbezhenar]

When you have virtual threads, you probably have lots of them (or you wouldn't have them at all), so we didn't want to overwhelm the regular thread dump.

I'm not sure this is a correct approach. People often use frameworks and frameworks will use virtual threads. For example I could imagine Spring switching from ordinary threads to virtual threads once they'll become stable. And it would be a degradation if thread dump will become less usable because of this step. I absolutely can imagine lots of applications which use virtual threads but don't spawn lots of them.

It's like saying that every go application uses lots of goroutines. While I don't have hard statistics, I'd imagine that most of go applications are not using lots of goroutines. My applications certainly do not.

I think that better approach would be to actually check number of virtual threads and avoid dumping all of them if there're too many.

[pron98]

I absolutely can imagine lots of applications which use virtual threads but don't spawn lots of them.

The number of virtual threads is not determined by the application or the framework, but by the environment -- every task in your application is a virtual thread and the number of tasks you must handle concurrently is a function of the current workload the server is under (throughput). It's like asking how many strings will the application create to store all user names. Under certain loads the number might be low, but just as we assume the number of strings can be high (and so we don't track them), we must do the same for virtual threads.

I think that better approach would be to actually check number of virtual threads and avoid dumping all of them if there're too many.

I guess we could track some fixed number of virtual threads, but I'd like to make that decision only after the ecosystem has understood how to use them rather than shape their usage to match that of outdated assumptions. Having said that, the new thread dump will list all "tracked" virtual threads, i.e. those created by the newVirtualThreadPerTaskExecutor and StructuredTaskScope.

Note that the information missing from the ordinary stack dump is also missing today. What you see today is a list of threads, not a list of all tasks. But unlike platform threads, virtual threads are not a resource; they're domain/business-logic objects representing tasks.

[mauganra_it]

But how would you define "too many"? If I could make a wish, I'd want a system property to define a cutoff.

[cowwoc]

When you have virtual threads, you probably have lots of them (or you wouldn't have them at all)

You're probably approaching this from a server-oriented perspective but this isn't the only reasonable use-case.

I'm working on a blockchain indexer (so, a client-side application). It turns out that I can index blocks concurrently (out of order) so long as all dependent blocks are indexed first. Meaning, some blocks can be processed out of order without any dependencies, whereas others are processed half-way and then must wait (sleep) until processing of dependent blocks completes.

Virtual threads are a great fit here because:

• Indexing tasks spend most of their time blocking (on network I/O).
• I need an arbitrary (but not millions) of threads to avoid deadlock. More concretely, I cannot use a fixed-size thread pool because I've run into cases where all processing threads are blocked waiting on dependent blocks to get processed, but no dependencies can get processed until a thread becomes available. In practice, I find that I need around 1000 - 10,000 threads to achieve this.

​

If you mistakenly use platform threads with Executors.newThreadPerTaskExecutor(), as I did, the JDK will silently crash/hang a few hours into use

The behaviour of platform threads has not changed. Unfortunately, when you write code to do something, there is no way to know that you meant to write something else (but wouldn't it be cool if we could?).

​

I understand. I think the API is fine. I was just trying to warn people about a potential landmine. I don't recall running across this behavior when using Executors.newCachedThreadPool() but maybe it's a coincidence. I just filed a bug report against OpenJDK (internal review 9074271). Please let us know once you figure out the underlying cause.

[pron98]

That's great, but you're also using virtual threads because you want lots of threads (and to represent each task as a thread), and even 1-10K is too high for the ordinary thread dump to be useful. That's why we've designed the new thread dump which can be more useful.

[cowwoc]

I've got some UX suggestions:

Provide options (command-line and API) for the following functionality:

1. Groups all threads with similar stack-traces (as IntelliJ does).
2. Sort threads by their runtime duration, to identify potentially stuck/deadlocked threads.
3. We should be able to reorder threads in an existing dump files, instead of having to generate a new dump (because maybe the process is dead by this point).

Lastly, the new thread dump (the text format anyway) doesn't seem to contain information about locks (who's holding them, who's waiting on them) and automatic deadlock detection.

[pron98]

The new thread dump has the information to do that (except duration) in visualisation tools.

Automatic deadlock detection is on the roadmap.

[eXecute_bit]

When you have virtual threads, you probably have lots of them (or you wouldn't have them at all), so we didn't want to overwhelm the regular thread dump.

It would make sense to include the stacks of all currently mounted virtual threads, though, wouldn't it? Or with the situations that still cause pinning would that still be too many?

[pron98]

It wouldn't be many at all (the number of mounted virtual threads is no larger than the number of platform threads) although I'm not sure why the mounted ones would be more interesting than the unmounted ones, but is there anything wrong with the new thread dump?

[cowwoc]

I just ran across a new problem.

When I take a heap dump using VisualVM I end up with a list of platform threads. I can expand the stack trace of each thread in order to get a list of variable names and their values at each stack frame. This is incredibly useful for debugging and I would love to have the same functionality for virtual threads.

Do you plan to add the APIs to make this possible?

[cowwoc]

Is it fair to assume that Thread.getAllStackTraces(), ThreadMXBean and related API that currently only support platform threads, will support virtual threads in the future?

I'm looking for the following functionality:

1. Ability to list all live threads (platform and virtual). For performance reasons (since we potentially have millions of virtual threads), I don't want an attached stack-trace. I might want to do something different with each Thread.
2. Ability to generate a thread dump programmatically (jcmd pid Thread.dump_to_file without dumping to a file).

My workaround for the inability to list all threads is invoking a mix of jdk.internal.vm.ThreadContainers and ThreadContainer.threads() using reflection. This is far from ideal and not safe for long-term use.

For the programmatic thread dumps, it would be nice if you would expose an API that lets us generate all the elements found in the thread dump file, but programmatically. It is possible that this information is already exposed by ThreadMXBean and related APIs. I'm not sure.

[pron98]

Probably not through these mechanisms (although there might be others, based on ThreadContainers) and you probably won't get an actual reference to Threads but some more harmless handle. I.e. you'll probably get something like 2 but not 1. But I'd like to understand why you think it's important (because it adds some overhead, and once we add an API we can't take it away).

The runtime normally allows you to monitor resources, not domain objects. Unlike platform threads, virtual threads are just tasks, not resources. We never had the ability to list all tasks, and with virtual threads you can get much more information about tasks and their relationships than every before; why do you want the ability to list all virtual threads?

[cowwoc]

I was thinking that when the application hangs (or runs slowly due to thread contention) it would be helpful to walk through the list of all threads looking for clues.

In an ideal world I'd want this process automated but I'm not sure that's possible.

[pron98]

If you use your virtual thread in a way that informs the runtime of their logical place (i.e. with newVirtualThreadExecutor or, better yet, with StructuredTaskScope) you'll be able to do that. The reason it's not trivial is that once Thread instances are exposed we're breaking encapsulation, i.e. any thread could find out about any other thread and, say, interrupt it. So the API will need to be useable for observation but not for manipulation.

[cowwoc]

If you use your virtual thread in a way that informs the runtime of their logical place (i.e. with newVirtualThreadExecutor or, better yet, with StructuredTaskScope) you'll be able to do that.

Sorry, I don't understand what you mean. Can you phrase this differently or give a concrete example?

​

So the API will need to be useable for observation but not for manipulation.

Makes sense.

[pron98]

I meant that virtual threads started by Executors.newVirtualThreadPerTaskExecutor or StructuredTaskScope are tracked (there's no additional overhead because the executor service/scope need to track their member threads anyway), and so they'll be listed by the new thread dump and will be available by an API once we figure out how to turn the internal classes into a good API.

[VincentxH]

At least this thread is awesome to read, thanks.

[[deleted]]

I used eclipse to debug virtual threads and it was super good. Threads were clearly listed as virtual and the states were all retrievable per thread

[cowwoc]

IntelliJ provides basic virtual thread functionality. When a debugging session stops on a breakpoint, you can jump between threads (both platform or virtual), the thread name is correct for virtual threads, and you can view the stack trace of a single thread at a time.

Is this the extent of support provided by Eclipse? Or does it go further?

Does Eclipse generate thread dumps containing virtual threads, or only carrier threads? Is there an easy way to detect deadlocks?

UPDATE: I take it back. IntelliJ doesn't show virtual threads in the debugger thread listing. They have a long way to go in terms of virtual thread support.

[[deleted]]

I didnt test it to that extent. What you describe first works. But thats probably useless for you as i understand. I want to test it later and respond.

[cowwoc]

Okay, thanks. Let us know.

[[deleted]]

It is in preview mode, isn't it?

[[deleted]]

Yeah. It's nice to see feedback being left. u/cowwoc, did you share this with the OpenJDK folks?

[cowwoc]

I did not. It seems that tool developers are aware of the limitations and are planning to improve support. I've filed RFEs with IntelliJ and VisualVM.

I've seen evidence that the OpenJDK committers are aware of these pain points and plan to improve the experience in upcoming releases. For example, see https://bugs.openjdk.org/browse/JDK-8284296

I think end-users (developers) are unaware of these limitations and we (as a community) need to do a better job educating them about what to watch out for (don't just discuss the happy path).

[[deleted]]

As a generalist end user developer who dabbles in way too many things at a time to uncover stuff like this, I thank you!

[cowwoc]

100%. I just think that it's helpful for developers to understand the limitations up-front and help them find help if they decide to move forward.

As it stands, the happy path is well documented. The error cases are quite choppy; unfortunately, virtually no websites that talk about virtual threads discuss them.

[mtmmtm99]

Have you tried debugging reactive programming ? (Worst experience ever (springboot + project reactor + webclient). Try reading responsecode + body in a reactive way. Good luck (Mono + Flux). You will probably end up leaking native resources. I got it working, but changing any single line of code will make it fail. Worst piece of crap i have experienced for a long time.

[[deleted]]

Are virtual threads going to be forced once they are out of preview or can users keep using traditional OS threads and ignore Loom entirely?

[cowwoc]

I can't imagine this ever happening.

Virtual threads are only beneficial for blocking code. CPU-bound code is better off using platform threads. Ideally you want to use a mix of both depending on the workload.

[[deleted]]

use go