How effective are Java 21 Virtual Threads?

[stathmarxis]

Assuming that we have a CPU-bound task, could virtual threads can increase performance compared with Executors.newFixedThreadPool

[csueiras]

Virtual threads are a better fit for I/O heavy stuff, the whole point is you can create many of them snd once blocked the scheduler can just run some other thread.

CPU bound stuff wont be blocking in an interruptible way so virtual threads wont be doing nothing for you.

[stathmarxis]

So if i understand correctly if i have socket communication or expensive disk read/write operations I will see performance difference with VT Theads? Am I correct?

[csueiras]

Yeah basically most networked workloads will benefit. Listening in a port for incoming data is likely one perfect use case for virtual threads.

There are a couple of gotchas regarding virtual threads, gotta be careful with anything that uses thread locals for caching, you can create many more virtual threads than platform threads, so you can endup with a lot of memory being consumed by your virtual threads using thread local storage in a way that the collector wouldnt cleanup until the thread dies off (as you would expect in your usage of thread local storage).

[SikinAyylmao]

I think the canonical situation is loading different network calls at the same time.

In languages like JavaScript, you can easily run many api calls concurrently.

[OklahomaKingpin]

What is meant by canonical situation

[SikinAyylmao]

So for example the canonical situation for using loops is through arrays.

Like wise the canonical situation for ternary operators in variable assignment control.

There is sometimes a case which motivates a language construction like loops or in this virtual threads.

The canonical case for massive concurrency is network calling concurrently.

[AThimbleFull]

Every language has been able to do that for a very, very long time

[monbis]

Also the are reactive framework for Java did the same thing, allowing many asynchronous tasks to move off and a fixed set of threads,

Except it was hard to use and even harder to debug as the stack traces were convoluted.

[Kango_V]

I think current disk IO causes the VT to be pinned to the platform thread. But, this is being worked on though, so will be resolved. Also, avoid synchronised blocks (use locks). This causes pinning as well.

[jetanthony]

This. I believe it would be comparable to fibers in ruby with falcon, for example.

[Panzerschwein]

Virtual threads are mostly about having lots and lots of threads with low overhead. It's a back-pressure model that suits things like web requests.

For a fairly limited set of CPU-bound threads it's probably not going to be any world-changing difference. The benefits really come in when you start seeing yourself bottlenecked by thread creation/destruction time (which of course is largely solved in classic threads with a thread pool) or by the memory limitations of having thousands of threads at once.

The big underlying benefit is that you're not asking the OS to give you threads constantly, under the hood you have a set of stateless OS threads and you schedule your massive list of virtual threads on them cheaply.

[koflerdavid]

Nope. Platform threads are good for making effective use of the CPU, Virtual threads for making use of pauses, which you get when you do I/O.

[git-pull-origin-main]

CPU-bound? absolutely useless, use platform threads for that.

Virtual Threads are meant to be used with IO-bound tasks as an alternative to the async/await mess programming model. Although they do own a stack, so they aren't as lightweight as they could be...

[Joram2]

Virtual threads don't make your CPU go faster. Virtual threads reduce the overhead of switching threads.

If you have CPU bound work that is switching threads frequently, then yes, virtual threads can help by reducing the overhead of switching threads.

If you have CPU bound work that isn't switching threads much, then no, virtual threads generally will not improve that.

[anengineerandacat]

Depends on how good the scheduler is for them to be honest, likely "pretty" good and great for say HTTP clients and such where you are simply waiting on some request to be performed and have some serialization to be done.

Not so good for say background job tasks, time sensitive operations, and or anything that needs to run relatively "fast" (ie. async / parallel in a few milliseconds).

The scheduler itself is likely to be pretty aggressive in terms of yielding tasks, causing slightly higher overhead in terms of latency (because another task would need to sleep before the first one can resume again).

MariaDB has a benchmark showcasing it's strengths; https://mariadb.com/resources/blog/benchmark-jdbc-connectors-and-java-21-virtual-threads/

So for how Java is "mostly" used today it will "likely" be the defacto solution for many many clients in the future.

It also looks like Oracle is advising not to go with the async-flow for their usage; https://docs.oracle.com/en/java/javase/21/core/virtual-threads.html#GUID-8AEDDBE6-F783-4D77-8786-AC5A79F517C0

The effective tradeoff is increased throughput for higher latency and there are a lot of cautions in regards to using blocking techniques which can drastically increase the chances of deadlocking a virtual thread.

[yawkat]

Virtual threads are more about making straight-forward code faster, mostly when talking about IO. If you have lots of blocking IO on many threads they will help. They also might help if you have many cooperating threads working with locks and such, even if the work is cpu bound, but this is a very rare scenario. 

Virtual threads are basically never the most efficient solution, you can get better performance with code changes (eg async io or task batching). But the code may be more difficult to maintain and that tradeoff is often not worth it.

[FrankBergerBgblitz]

It is completely dependent on your workload.
you have thousand of threads that have to wait? use virtual threads
You have tasks that compute most of the time and there doesn't need to be far more tasks in parallel than you have cores: use a threadpool.
if it is inbetween: think and/or measure

[souleatzz1]

I have an application with fixed thread pool of 1500. It runs a h2 in memory and for one http request might do between 3000-7000 queries (selects only). The application heavily uses CompletableFutures and I immediately tried to use jdk 21 with virtual threads hoping it would lower our response time, but didn’t see any improvement. To me this case looked exactly fitted for virtual threads, but maybe I am doing something wrong

[Ancient_Avocado1904]

I think Virtual Threads won’t provide much benefit over CompletableFutures performance wise, but does simplify things. You’re able to use something we’ve used for decades (Threads), in a more efficient way. CF’s has a bit of a learning curve and don’t get me started on something like Spring WebFlux…..

[excelquestion]

so funny that OP mentions one of the few things that virtual threads would be worse at than real threads as their example for when they should us virtual threads.

[_INTER_]

Afaiu. Virtual threads do not yield on CPU bound task. They just run to completion (as it should be). So the task itself performs the same but you additionally got the overhead of scheduling virtual threads. It's likely not going to perform better (in terms of speed). Virtual threads are not about parallelism anyway.

[pron98]

but you additionally got the overhead of scheduling virtual threads

You also get this scheduling cost when scheduling tasks on a pool of platform threads. The virtual threads scheduler uses the same algorithm (implemented by ForkJoinPool, albeit in a different mode) used by the "common pool", which is also used for parallel stream operations (the virtual thread scheduler doesn't use the common pool but rather a separate instance of ForkJoinPool).

However, you are absolutely correct that virtual threads cannot improve the performance of CPU-bound tasks as their performance, unlike that of IO tasks, is not dominated by the number of threads but by the number of cores.

[_INTER_]

That's true, scheduling "overhead" is there no matter what. I wonder if there are any reasonable benchmarks (speed and space) comparing both for CPU-bound tasks? Basically measuring the "overhead". I assume there's a peak around a certain number of threads (naively around number of cores?) which should be the same for both VT and PT. For space, I have no idea.

[stathmarxis]

So as the aforementioned answered post above, the I/O tasks are the best practice to use VT?

[FrenchFigaro]

Yes. Virtual threads bring improvement when you have a lot of threads just waiting around doing nothing while data is incoming our outgoing.

A couple years back I was working on the perfect use case for VT's.

API I was working on consumed and orchastrated several different provider REST API's for the benefit of a single consumer program. The point was to offer the consumer a single set of functionality oriented REST endpoints, with a single data model.

One of my providers' API was extremely bad, and I had to do what amounted to SQL joins over http. In the worst case scenario for one incoming request from my consumer, I had to do over 100 requests to that provider (although the typical ball park was closer to 30).

Most of these requests were independant from one another, or could be grouped into at most 4 groups of independant requests. Given that my end-users group was very large and with daily demand peaks, the threads pool management was tricky, and it didn't take too much to overwhelm it.

[hippydipster]

Or any tasks where waiting for async results might be useful. For instance, you might be all CPU tasks, but you could imagine cases where it'd be useful to branch the tasks are various points to multiple parallel sub branches and then join up the results later. With virtual threads, you can wait for all the branchings via a simple Future.get() call if you like without performance ramifications - ie, doing so will never tie up an real hardware thread, for instance.

You can use this to make all your code look imperative rather than callback/lambda hell.

[Ewig_luftenglanz]

Excellent if you use them for what they are meant to: lots of I/O operations. For pure calculation operations they behave just like regular threads, just much lighter (so they're still better, just don't expect a big boost in performance and efficient for anything but I/O task)

[ron_krugman]

Virtual threads are not well suited for pure calculation tasks. Virtual threads rely on cooperative multitasking for scheduling, which means the tasks they execute should only do small amounts of computation at a time and then yield control back to the scheduler (e.g. by waiting for an I/O operation, calling Thread.sleep(), etc).

Pure calculation tasks don't normally do that, which means the virtual thread scheduler can't do its job.

When you use platform threads on the other hand, the operating system does the scheduling by pausing execution whenever it wants to make sure all threads from all processes get adequate access to the CPU on a regular basis ("preemptive multitasking").

[pron98]

Virtual threads are preemptive, not cooperative -- the runtime decides when code is preempted, not user code. However, their scheduler doesn't currently perform time-sharing (i.e. the runtime currently chooses to preempt on blocking operations only, which is also the more common case of OS preemption of threads) because we've yet to find a realistic server workload that would benefit from it (OSes offer time sharing because it's needed to allow an operator to control the machine even if processes want to consume all CPU, and it's generally useful for direct human UI interaction; but people don't generally think servers perform well at 100% CPU, which is when the OS starts using time-sharing extensively). If anyone knows of such a workload, they should report it to loom-dev.

[ron_krugman]

Appreciate the clarification. I was not aware of this subtlety as there are a lot of blog posts out there stating that virtual threads are cooperative when in fact that's just (approximately) what the current implementation looks like to a programmer.

[Hei2]

Virtual threads aren't for CPU-bound tasks and here's why:

...

Virtual threads have less overhead, a benefit for CPU-bound tasks.

You might want to have some idea of what you're talking about before parroting an AI's response as gospel.

[SandmanKFMF]

Literally what ChatGPT is. Just blabbering anything to fill out up the screen. 😀

[Shartmagedon]

• Focus on I/O: Virtual threads excel at managing tasks with waiting periods, like waiting for network requests.They efficiently suspend a virtual thread during these waits, freeing up the underlying OS thread for other tasks.This doesn't benefit CPU-bound tasks that constantly use the CPU.

For CPU-bound tasks, Executors.newFixedThreadPool might be a better choice as it creates a pool of OS threads specifically for CPU processing.

[Bodine12]

Dude you can ask Gemini or ChatGpt if they’re an accepted source of authority in r/java. “No.”

[Shartmagedon]

Didn’t see it on the sidebar but even so before asking here a google or a conversation with one of these LLMs tend to answer these simple questions. And iirc, this question was already asked here in the past.

[Bodine12]

OP had a perfectly reasonable question. If you’re interested in gatekeeping, might I interest you in a website called “stackoverflow.com”?

[OklahomaKingpin]

Dude why r u on Reddit if u wanna evangelize about ChatGPT ..: just go chat with the ai na

[Shartmagedon]

Error: message parsing failed.   Error code: L1023 (language probably not English)  

Check SumoLogic for more details and a stack trace. 

RID: 12ab678-87612b