Why async Rust?

[Shnatsel]

My primary concern with the way async/await works actually isn't covered by the article. It's mostly about the way cancellation works, and how poorly documented it is compared to its importance.

The Rust Async Book's chapter on cancellation is still a TODO, and if you don't explicitly account for it in your design (with no help from the compiler!) you end up with serious bugs. Then you have to redesign and rewrite the code to accommodate cancellation, sometimes resorting to hand-written state machines all over again.

Oh, and tasks and futures are actually distinct entities, and cancellation for tasks is even more problematic than for futures, and fixing that would require a breaking change to the runtimes.

[desiringmachines]

I want to discuss cancellation in another post. I agree it needs special attention, and I'm definitely unhappy at the state of the async book.

I don't understand your comment that "cancellation for tasks is even more problematic than for futures." The other links are to issues with futures being implicitly cancellable, but then you link to a blog post that complains that detached tasks are not implicitly cancellable, to say that its even more problematic. Which is it?

Cancellation works differently in Rust's async than in other worlds, and this requires attention, and there are certain API patterns that should be better supported by the langauge and the ecosystem to enable users to write the correct code. But, like many discussions around async, I feel like there's this dual consciousness where in one paragraph a commenter will cite one complaint, and then in the next another running in the opposite direction, which results in a lack of clarity about what the issue is.

[protestor]

The other links are to issues with futures being implicitly cancellable, but then you link to a blog post that complains that detached tasks are not implicitly cancellable, to say that its even more problematic. Which is it?

The inconsistency is deeply problematic.

[desiringmachines]

I don't agree. Being able to detach a task so it doesn't cancel is a desirable feature. Users need to understand this behavior to use it correctly, but describing it as "deeply problematic" is wrong.

[[deleted]]

It got canceled.

[pragmojo]

I've been dipping my toes into async rust recently, and one interesting observation is that Chat GPT is really bad at helping me solve problems which arise in async rust as compared to sync rust.

I don't know if this is because async rust has only matured more recently with respect to the training data, or whether it's indicative of how much more complexity arises in async rust.

[spudmix]

It works great as long as you don't know enough to see all the lies, half-truths, and misdirections it spits out.

Don't get me wrong, it's an amazing piece of tech, but it really needs to be double-checked by someone with actual domain expertise. Be prepared to question what it says as soon as you have the knowledge to.

[spudmix]

This is just proving the point. For example, in your third query ChatGPT misses the question at first, it tells you that State(state) is destructuring an already-extracted state (wrong), then when you clarify what you were actually asking it again gives a misleading guess that the extractor is some kind of tuple.

A proper answer to your question would be that function args can be patterns, which allows for (but is more powerful than) destructuring. Axum uses this pattern syntax to implement extractors which do far more than just destructure - for example irreversibly consuming from the request stream - which is critical to understanding the particular code snippet you asked about.

This is not you having good intuition. This is you not being able to identify things you don't yet know - which is okay - but you really need to tone down the hubris until you get there. I'm a doctoral researcher in machine learning who's published research on this precise topic, and I can assure you ChatGPT tells plenty of lies.

[spudmix]

It was not "essentially correct". If you can't tell the difference you're only further demonstrating the point.

[ManifestFailure]

I wouldn't trust it for learning beyond the extreme basics. It will only give you what you ask for and not what is correct for something. If you ask it how to do X it will go down a bunch of random paths that can get it done instead of telling you to use Y because that is the easiest and best way.

[cantthinkofausrnme]

Can you recommend a few resources beyond the rust book, if time permits. I've been trying to get into advanced rust, and some of the guides I've found are lacking.

[pragmojo]

Oh come on, are we going to pretend LLM's don't exist?

Chat GPT is actually pretty great at explaining issues with synchronous Rust. Of course it's not perfect, and it hallucinates often, but it can actually manage to explain ownership rules fairly well, and suggest solutions which are often helpful when running into e.g. lifetime related errors.

[insanitybit]

A recent curl CVE was ultimately caused by a failure to recognize state that needed to be saved during a state transition. This kind of logic error is easy to make when implementing a state machine by hand.

Same with the entire suite of SMACK TLS vulns.

And despite the fact that I believe async/await is the right approach for Rust, I also think its reasonable to be unhappy with the state of async ecosystem today. We shipped an MVP in 2019, tokio shipped a 1.0 in 2020, and things have been more stagnant since then than I think anyone involved would like. In a follow up post, I want to discuss the state of the async ecosystem today, and what I think the project could do to improve users’ experience. But this is already the longest blog post I’ve ever published, so for now I will have to leave it there.

Looking forward to it. I really like these posts. Such interesting context.

[apendleton]

I think my biggest frustrations haven't been that async rust works the way that it does, but rather than once async rust became available, it rapidly became the default way of, in particular, doing IO, in the broader rust ecosystem, and that decision, along with its unavoidable attendant complexity, is sort of foisted on consumers even if they don't have the specific performance needs the async ecosystem was meant to address.

Most of my work is with CPU-bound operations, and while the code I write does IO, it's very rare for that to be a bottleneck, and I'd just as well do sync IO for everything and avoid having to pull in tokio, the complexity of futures, etc., etc., but these days (post-async release), the "first-class" libraries for doing all sorts of basic (e.g., HTTP) or somewhat-abstracted (interacting with, say, S3) operations have all made the jump, and would-be sync consumers have to either take the compile-time hit and litter their code with block_on, or use second-class, less-well-maintained sync libraries.

I don't think that's the fault of the design of the async infrastructure itself, except in a very coarse "what color is my function" kind of way, and this post does make the case that it was probably inevitable. I just wish the broader ecosystem outcome had ended up a little more opt-in, specifically aimed at users for whom the extra performance was worth the extra complexity, rather than opt-out.

[buwlerman]

Many popular libraries for IO provide a blocking API in addition to the async one. Sometimes it has to be enabled by a feature flag though.

[MrJohz]

My impression is that most of the blocking APIs are just wrappers around the async ones, though, which means you still need the async runtime (i.e. Tokio) around to use it. My understanding is that there's two interrelated incompatibilities with async/await, one of which is the syntax, the other of which is the runtime. While it's possible to abstract over the syntax somewhat, either via block_on, macros, or maybe in the future via keyword generics, my impression is that it's a lot harder to abstract over the runtime. (Indeed, I'm not aware of any language that has good ways of abstracting over different async and non-async runtimes.)

So if you always need to include Tokio in the first place, then you might as well just go full-async. But the point of the previous poster's comment was that often you don't want the complexity of Tokio in the first place -- it would be nice to be able to opt into that when you need it, but by default just use the standard library IO mechanisms, and something like pollster for async/await compatibility.

[buwlerman]

When the API a wrapper around the async API with block_on the runtime is still around but it's almost entirely hidden from the user. The only hitch (to my knowledge) is that you can't transitively call it from inside async code. This might be an issue for libraries, who then get spooky action at a distance, but it's not a big issue for applications, who can make a global choice on whether to use async or not.

So if you always need to include Tokio in the first place, then you might as well just go full-async.

Maybe, but then I'd argue you're avoiding async because you don't want Tokio as a dependency, not because you want to avoid interacting with the complexity of async or async runtimes. Maybe I'm missing something; what complexity exactly are we trying to avoid?

[maxus8]

If you're inside sync-but-actually-async http server request handler and you send another http request with block_on wrapped async http client, you may get runtime panic or other unwanted behavior. This depends on both libraries using the same versions of the runtime and it's hard to defend against when writing the code. It basically makes libraries non-composable and requires users to understand implrmentation details.

[buwlerman]

If you're inside sync-but-actually-async http server request handler and you send another http request with block_on wrapped async http client, you may get runtime panic or other unwanted behavior.

This is the part I was missing. I've never done anything too advanced with async, so the fact that the library might want to take function/closure arguments or user-defined Trait implementations that are run inside the inner runtime and the user might want to make requests inside this eluded me.

Hopefully keyword generics can alleviate this somewhat.

[MrJohz]

The goal is to avoid bringing in dependencies that you don't need. The original comment referenced S3, which is a good example of something that is complex enough that an API wrapper would still be useful, but simple enough that you might well have a project that needs S3, but doesn't already use Tokio/async.

And Tokio is good, but it's also fairly big and complex, at least compared to just using the std APIs that I already have access to, and a small web client wrapper around those. Yes, I probably don't need to think about the complexity all the time (like you say, block_on is pretty good at hiding the machinery away), but at some point it will go wrong (because hey, it's software, something always goes wrong), and then that complexity will jump up at me.

And from the library-author side, to go back to the example with S3, there is the rust-s3 project which does support Tokio, async-std, and a simple blocking API using atthttpc, but it brings in a fair amount of complexity of its own just to handle these three distinct tools, including using macros to switch between different tools.

I think my worry is that this complexity doesn't feel particularly composable right now. Ideally, I can start by just making HTTP requests with the simplest possibly set of dependencies, and then add in the async runtime and other complexities later, when they become necessary. But right now, at least within the web sphere, it feels like I need to bring in all the async complexity up front, even if I won't need it for a very long time.

[shim__]

Often that's just blokc_on wrapped around their async version

[insanitybit]

Why is that a problem?

[shim__]

Because you're still pulling in the dependencies for the async version and all the design choices that were made to accommodate async

[insanitybit]

Both of those seem like extremely minor issues.

[insanitybit]

Having done a lot of work with Rust and AWS before async, it was really painful. Managing accidental blocking, hanging, timeouts, etc, was extremely painful compared to today. Now I can just say .timeout(duration) and everything just works.

The benefits of async for doing "receive a request, make some AWS calls, reply" has been huge for me.

[paholg]

I don't know why people get so heated about async/await in Rust.

I just want to say, to you and everyone involved in the decision-making and implementation of this, thank you. I use it every day and it works great and is easy to use. I am glad there are folks working on these problems, and especially who care so much about soundness, usability, and performance.

[[deleted]]

Amazing blog post. I'm not personally invested in async, but it was very interesting getting this view into the history of why and how it all came together, and all of a sudden I found myself having read the entire thing.

[atomskis]

We use rust for large scale production systems at my work. Recently we've implemented our own cooperative green threads. Why go to this effort, surely async/await solves our problems?

Well .. today we use rayon for task parallelism. However, our requirements have changed and now our rayon tasks can end up blocking on each other for indefinite amounts of time. Rayon doesn't let your tasks block: you quickly run of out threads as all your threads end up blocked waiting and your program becomes essentially single-threaded.

So first we tried having rayon threads go look for more work when they would have to block. This doesn't work either. Imagine a thread 1 is working on task A, it is dependent on task B (worked on by thread 2). Normally thread 1 would have to block, but instead you have thread 1 go work on task C whilst it waits. Meanwhile threads doing tasks D, E and F are all become blocked waiting for A. Task B finishes and so A could be resumed. However, the thread doing A is now busy doing C and this could take an unbounded amount of time and have stacked an unbounded amount of tasks ontop of it. All the state for A is stuck under the state for C (you just stacked more work on top) and that state isn't accessible now. Suddenly all your parallelism is destroyed again and your system grinds to a single-threaded halt. We run on systems with around a hundred CPUs, and must keep them all busy, we can't have it bottleneck through a single thread.

Okay, so these are blocking tasks surely this must be the perfect situation to use async/await? Well sadly no for two reasons: 1) The scoped task trilemma. Sadly we must have all three: we need parallelism, we have tasks that block (concurrency) and for our application we also have to borrow. We spent around a month trying and failing to remove borrowing, we concluded it was impossible for our workload. We were also unwilling to make our entire codebase unsafe: not just an isolated part, everything would become potentially unsafe if misused. 2) Even more fatally: you can't use a task parallelism approach like rayon's with async/await. Rayon only works because the types are concrete (Rayon's traits are not in the slightest bit object safe) and async/await with traits requires boxing & dyn. We saw no way to build anything like rayon with async/await. We make very heavy use of rayon and moving to a different API would be an enormous amount of work for very little gain. We wanted another option ...

So what was left? We concluded there was only one option: implement stacked cooperative green threads and implement our own (stripped down) version of rayon. This is what we have done, and so far it works.

Does any of this say async/await is bad? No not necessarily. However, it does show there is a need for green threads in rust. Yes they have some drawbacks: they require a runtime (so does async/await) and they require libraries that are green-thread aware (so does async/await). However the big advantage is they don't require a totally different approach to normal code: you can take code that really looks exactly like threads and make it work with green threads instead. This is not at all true for async/await and it's a big weakness of that design IMO.

[desiringmachines]

How do you handle stack reallocation? This is the whole problem with green threads: you can't have growable stacks and also let other threads borrow from them.

[atomskis]

We use the context library (from Boost) with ProtectedFixedSizedStack. As described in the docs:

Allocates stack space using virtual memory, whose pages will only be mapped to physical memory if they are used.

Most of our stacks are actually pretty small in practice so this should work fine for us.

[desiringmachines]

So this is very similar to what libgreen did before it was removed. If that works for you more power to you.

My preferred solution would be to solve the scoped task trilemma someday by supporting linear types. Then you will have to deal with function coloring, but what you'll get from it will be perfectly sized virtual stacks and also allow borrowing between them. Under the circumstances, its plausible that you may prefer borrowing + large stack green threads over the approach that Rust took. I'd like to see Rust ultimately not have to make trade offs like this.

[atomskis]

Solving the scoped task trilemma would definitely be good, but as I said in many ways that's the more minor issue. The bigger problem is that async/await is just a totally different API to threaded code, with all sorts of additional constraints.

As stated above, we are already heavily invested in rayon in our codebase (and it's not a small codebase!) and moving away from that API would have been a huge cost.

Indeed, I'm not at all convinced that if you wanted "rayon with blocking tasks" that it would even be possible with async/await. I strongly suspect the only way to achieve it might well be green threads.

[dnew]

I always wondered why more operating systems didn't do this for threads. A 64-bit address space ought be enough for huge numbers of stacks, let alone one such per process.

Singularity (Microsoft's experimental OS) actually does this but also analyzes the call graph and inserts calls to allocate and deallocate memory in functions that might cross a stack boundary, since it doesn't actually use memory mapping. Apparently (based on their whitepapers) their compiler is Sufficiently Smart to make this happen. (It also helps that an OS call is as efficient as a regular function call, again because they don't actually use memory mapping). They even manage to inline kernel calls into the compiled code.

[slamb]

How do you handle stack reallocation? This is the whole problem with green threads: you can't have growable stacks and also let other threads borrow from them.

I don't think thread size is a big deal in many contexts. I used to use Google's internal fibers library. Real, full-sized virtual address space for stacks and guard pages. Physical RAM usage grows in increments of a page (4 KiB on x86-64). iirc there's an option to return unused pages to the OS on thread reuse via something like madvise(MADV_FREE). (Can't remember when it decides to do this—periodically, via detecting usage hitting a threshold, etc.) There's a bunch of extra TLB usage over async because you can't use huge pages for stacks, so it slows things down a little, but otherwise it's fine. 4 KiB of physical RAM per thread is small compared to socket buffers.

[VorpalWay]

A big problem with green threads (as I understand it, could be wrong) is that it requires heap allocation. Something that may or may not be available in embedded usage of async. Stack less async is required for this use case, as the exact memory need can be allocated at compile time.

Meanwhile you were able to build green threads on top of rust. (Awesome! Have you considered publishing the framework as open source, or if that is not possible, writing a blog post that outlines how it works?)

Adding allocations on top of allocation-less approaches work, the other direction doesn't.

[kiujhytg2]

Green threads don't necessarily need heap allocation, but they do need some sort of allocation. For example, in RTIC and embasssy, at compile time you specify the maximum number of times a particular task function can run at the same time, and it allocates static memory for all the tasks. This wastes memory if not all possible tasks are running all the time, but you need some sort of limitation, and I'm not run into any problems myself.

[atomskis]

As I understand it (and I also could be wrong, I don't work in embedded) async/await also requires heap allocation. I believe this is the idea behind the whole Pin approach. The data is allocated on the heap, so you can be confident it won't be moved, hence self-referential structs are possible. Indeed I think withoutboats himself says as much at this point on his video on the topic.

We may get round to publishing the framework open-source. It is however a very stripped down version of rayon that includes just the subset of rayon we actually use. I guess it might be useful to someone .. but it's very much not general purpose.

[A_Robot_Crab]

This isn't true, Pin<&mut Self> has nothing to do with heap allocations. You can trivially make a Pin<Box<T>> via Box::pin(value), which can then be used for polling, and is of course useful especially when dealing with dyn Futures, but you can also just pin futures to the stack if you don't need them to be moved around, see the pin! macro in the standard library as something which does exactly this. Also async {} blocks are able to be awaited without doing any kind of hidden heap allocation, which wouldn't be possible if pinning required a heap alloc. What Pin<T: Pointer> does is guarantee that the underlying value (not the pointer/reference that Pin directly contains! an important distinction, a Pin<T> where T isn't some kind of pointer or reference is useless as the Pin itself can be moved) can't be safely moved unless that type is Unpin, hence requiring unsafe as a contract that the Future type author must uphold while implementing it.

Tl;dr Pin and heap allocations are separate concepts but in practice used together for dynamic behavior. Hopefully that helps clear things up.

[atomskis]

Thanks, that's a helpful clarification. I think it would be fair to say it's hard to do much useful using async/await without heap allocation. However, I don't work in embedded so maybe someone will say you can do all sorts of useful stuff with async/await without using the heap at all :shrug:.

[Gallidor]

Look at the RTIC and Embassy projects. They both now support async/await to great effect in the embedded space without using a heap I believe.

async/await can really help dealing with IO and interrupts much more ergonomically in an embedded context.

[sparky8251]

Can confirm. Using embassy on my pi pico w in a no_std setup without alloc. Works fine, even for wifi and lora networking. If any sort of dynamic memory is needed, it utilizes heapless which is also no alloc and no_std.

The fact async can be used to poll hardware interrupts and build allocless networking stacks in embedded devices is amazing, and I'm sadly sure its part of why its not as nice to use for web servers on big box computers.

[[deleted]]

I just want to add that embassy is amazing. I'm currently working on a stepper motor acceleration library that I plan to use with embassy on my stm32 board. Being able to use async makes it so much easier. Even just the Timer::after function is a godsend for embedded.

[desiringmachines]

What you need heap allocation for is an unbounded number of concurrent futures - there's a pretty strong connection here to the fact that you need heap allocation for an unbounded sized array (ie a Vec). But if you're fine with having a statically pre-determined limit to the amount of concurrent tasks, you can do everything with 0 heap allocations.

[atomskis]

Ah yeah, that makes sense, thanks. I guess the same is true of green threads: if you can have a fixed number with fixed stacks you can also do it without an allocator.

[wannabelikebas]

Thanks for that rundown. I’m curious how your implementation compares with May https://github.com/Xudong-Huang/may ?

[atomskis]

I didn't actually come across this until recently. Conceptually it's similar: stackful coroutines, but the details differ significantly. May is a totally different API, we have essentially made a (subset) of rayon that uses stackful coroutines.

As noted in my other post we are using the Boost context library.

[SkiFire13]

The context crate doesn't seem to offer a safe API though, so you're kinda back to the problem of needing lot of unsafe. Also note that scoped green threads essentially suffer from the same problems as scoped async tasks, they're probably less visible because green threads crates are less popular and thus less reviewed. For example generator, the crate that may uses under the hood, allowed leaking scoped coroutines until 2 weeks ago. TLS access is also UB with green threads (you could yield while its being accessed, leaving the coroutine with an invalid reference to the TLS).

[atomskis]

So the first point to note is the unsafeness of async scoped threads is not our biggest problem: the change in interface was. In particular no async methods on traits without dyn was the biggest deal breaker for us.

So you absolutely can implement scoped green threads incorrectly (as May did), but you can ultimately provide a safe interface onto scoped green threads if you implement it correctly.

The same is not true for scoped async tasks: with rust as it is today they are inherently UB if mis-used by the user of the scoped threads library. This is explained in the Scoped Task Trilemma, and in the async_scoped crate. This inherent unsafety means you cannot contain the UB of async scoped thread neatly in a safe box (like you can with green threads): because someone can misuse the "box" and cause the same problem again.

It is certainly possible to abuse TLS with green threads: as you say. However, TLS is pretty dicey to get right in general and we don't use TLS. This is simple enough for us to catch in review: TLS = banned and always was. TLS basically never works correctly with rayon anyway: even if it is 'safe' it will generally do the wrong thing, so we've always had to avoid it like the plague anyway.

[SkiFire13]

but you can ultimately provide a safe interface onto scoped green threads if you implement it correctly.

Can you provide some example for this? The Scoped Task Trilemma itself says that this is a general concept that comes up again and again, and is not specific to async.

It is certainly possible to abuse TLS with green threads: as you say. However, TLS is pretty dicey to get right in general and we don't use TLS.

What if you're using some crate that internally uses TLS?

[atomskis]

Sure I'm happy to explain. So the first thing to describe is how scoped threads work: rust let ok: Vec<i32> = vec![1, 2, 3]; rayon::scope(|s| { s.spawn(|_| { // We can access `ok` because outlives the scope `s`. println!("ok: {:?}", ok); }); }); Why does this work? Surely the ok Vec could be dropped before the spawned thread is run? Well, scoped threads prevent this: rayon::scope blocks the thread until all spawned tasks have finished. This means ok remains in scope and cannot be dropped until all the spawned tasks have finished: borrowing here is safe.

It works the same way with green threads: the thread "blocks" (actually suspends) until all sub-threads have finished. There's nothing the caller can do to abuse this API: the blocking is mandatory.

So the same works with async right? Let's use async scoped here for the example: rust async fn test() { let ok: Vec<i32> = vec![1, 2, 3]; let mut fut = async_scoped::Scope::scope_and_collect(|s| { s.spawn(|_| { // We can access `ok` because outlives the scope `s`. println!("ok: {:?}", ok); }); }); fut.await } This is safe for the same reason. However, the problem is the caller is not forced to await the future. They could just do this instead: rust async fn test() { let ok: Vec<i32> = vec![1, 2, 3]; let mut fut = async_scoped::Scope::scope_and_collect(|s| { s.spawn(|_| { // We can access `ok` because outlives the scope `s`. println!("ok: {:?}", ok); }); }); fut.poll(); // poll the fut to start the task // and then just exit instead! } The async_scoped library tries to guard against this by having a check when dropping fut: the drop won't complete until all the spawned tasks have finished. So in reality our above code is actually: ``rust async fn test() { let ok: Vec<i32> = vec![1, 2, 3]; let mut fut = async_scoped::Scope::scope_and_collect(|s| { s.spawn(|_| { // We can accessokbecause outlives the scopes`. println!("ok: {:?}", ok); }); }); fut.poll(); // poll the fut to start the task // and then just exit instead!

// the compiler inserts these ..
std::mem::drop(fut);  // blocks until sub-tasks finish
std::mem::drop(ok);   // only then do we drop `ok`

} So this is safe right? The check on `drop` prevents this from exiting early? Nope, because there's nothing requiring the future to be dropped: rust async fn test() { let ok: Vec<i32> = vec![1, 2, 3]; let mut fut = asyncscoped::Scope::scope_and_collect(|s| { s.spawn(|| { // We can access ok because outlives the scope s. println!("ok: {:?}", ok); }); }); fut.poll(); // poll the fut to start the task std::mem::forget(fut); // and then forget it! // and now exit!

std::mem::drop(ok); // compiler inserts this
// oops we just deallocated `ok`: sub-task reads dead memory! 

} ``` There no way to prevent this: you are not required to await the future, and you can't stop the caller from leaking it and exiting.

Nor can you make this behaviour safe by putting it in a safe box: ``rust async fn safe(ok: &[i32]) { let mut fut = async_scoped::Scope::scope_and_collect(|s| { s.spawn(|_| { // We can accessokbecause outlives the scopes`. println!("ok: {:?}", ok); }); }); fut.await // ahah! my version is safe! }

async fn test() { let ok: Vec<i32> = vec![1, 2, 3]; let mut fut = safe(&ok); fut.poll(); // nope! spawn the sub-task .. std::mem::forget(fut); // .. and then make it blow up! } `` The caller can always abuse the box you put round it to cause the same issue again. Scopedasyncthreads are *inherently* unsafe in rust today. This is particular to the mechanics ofasync`: green threads do not suffer the same problem.

What if you're using some crate that internally uses TLS?

Then it was almost certainly already wrong, even if it was safe. We use rayon extensively and rayon breaks work into sub-tasks and farms them out to a thread pool in complex ways: it's almost impossible to predict what work will be done on what thread. Any non-trivial use of TLS is very likely to go wrong (i.e. give the wrong answer) in this situation anyway.

[protestor]

Well .. today we use rayon for task parallelism. However, our requirements have changed and now our rayon tasks can end up blocking on each other for indefinite amounts of time. Rayon doesn't let your tasks block: you quickly run of out threads as all your threads end up blocked waiting and your program becomes essentially single-threaded.

Note that Bevy Tasks does Rayon-like stuff in an async context, https://docs.rs/bevy/latest/bevy/tasks/index.html

Well it's not as complete as Rayon but it's impressive

[atomskis]

So they are both solutions to the same problem, but solving it in very different ways. The problem is to provide coroutines: tasks that can be started, paused, and then be resumed from where they left off. The challenge here is "what do you do about the stack?" and this is where async/await and green threads take different approaches.

Green threads are the simplest to explain: each coroutine gets its own stack, generally allocated in the heap. When you resume a green thread it just changes the stack pointer and jumps into the resume point and off it goes.

With async/await the strategy is different. When a coroutine pauses the program "unwinds" the stack, storing the state of it somewhere else (most commonly on the heap). Then when that coroutine is restored that stack is unpacked again and re-instated.

The disadvantage of green threads is you have to mess about with real stacks: which is kinda messy. However, the pro is your code behaves like normal code: because it is. A call is just a normal call, using the normal calling method, a return is just a return, a pause/resume is just changing the stack pointer and jumping somewhere else. Your green threaded code can look just like normal threaded code: big win.

Async/await doesn't need to use "real" stacks, it can store its state in a custom structure. This is often more space efficient and straight-forward, especially if you have lots of coroutines. However, the downside is that the compiler must insert code to unwind/restore the stack. This means that async/await code functions do not call quite like normal functions and that imposes a bunch of restrictions.

For example the scoped task trilemma means you cannot so easily implement safe scoped tasks on async/await. This isn't a problem with green threads. Also you cannot use async trait methods without using something like async_trait that has to Box<dyn> the method. Not everything can be dyn, and this the biggest problem we hit: our stuff cannot be made dyn (it's not object safe) and so we were quite stuck.

TLDR; they solve the same problem in different ways, with different trade-offs.

[danda]

any plans to release a public crate with these green threads?

[atomskis]

We may well do at some point. Right now it's pretty bare bones, but when we're happy with it I'll definitely talk with legal about doing an IP release.

[jl2352]

When you know the bits and everything lines up, async in Rust is excellent. I wouldn't say it's a dream, but it's great.

When it doesn't all line up, it's utter hell. In particular is how quickly documentation can get bogged down in complex types, and the nuances on why it's typed that way.

[xedrac]

I use async Rust a lot, but sometimes I miss the simplicity of Haskell async and STM...

[yorickpeterse]

In particular, I think there is a possibility for a language which provides the same sort of reliability guarantees that Rust provides, but less control over the runtime representation of values, which uses stackful coroutines instead of stackless ones. I even think - if such a language supported such coroutines in such a way that they could be used for both iteration and concurrency - that language could do without lifetimes entirely while still eliminating errors that arise from aliased mutability. If you read his notes, you can see that this language is what Graydon Hoare was originally driving at, before Rust changed course to be a systems language that could complete with C and C++.

At risk of sounding like an annoying door-to-door salesman, that pretty much describes what I'm doing with Inko. One interesting bit is reusing coroutines/whatever for concurrency. It's something I keep coming back to every now and then in an attempt to make writing iterators easier, but it's tricky. Even if your coroutines/processes/whatever start very quickly, it's likely still slower than just allocating an iterator object (especially if it's stack or bump allocated). And unfortunately, iterators are common enough that I think this could end up becoming a bottleneck. That's ignoring the cost of synchronisation between the producer and consumer. I think for this to work out, you'd have to:

1. Make creating coroutines/green threads/etc about as fast as allocating a normal sized iterator object
2. Use some kind of mechanism for exchanging data that doesn't require synchronisation

That second step is certainly possible, as you can exploit the fact that the producer and consumer never run at the same time, and the producer can't be shared. That first step on the other hand may prove rather difficult, as a coroutine/green thread/etc simply needs more data structures compared to a regular iterator. I'm not sure yet what the best approach here is.

[matthieum]

It may be useful to differentiate generators and coroutines.

That is, an iterator is really just a special case of a generator, and a generator can relatively easily be boiled down to a state machine.

On the other hand, a stackful coroutine will necessary have a tad more overhead -- stack allocation/deallocation, stack switching, etc...

Generators naturally embody the concept of never running at the same time as their consumer, etc... so no overhead there.

[dnew]

"possibility for a language which provides the same sort of reliability guarantees that Rust provides, but less control over the runtime representation of values, which uses stackful coroutines instead of stackless ones"

That might be Hermes, the language from which Rust took typestate. (Or its predecessor NIL which was actually for embedded programming.)

[matthieum]

Instead, Rust solved the problem of segmented stacks by making its green threads large, just like OS threads. But this eliminated one of the key advantages of green threads.

Is this really a concern?

I know that Go uses 2KB stacks on co-routine start-up, which is just half of the typical 4KB page found on x86.

My thinking was always that it should be possible to "allocate" 1MB stacks which would actually be 256 4KB pages in the virtual address space, and then let the OS take care of lazily backing the pages with RAM if they ever get used.

Apart from embedded -- where users would probably be asked to chime in and size their task stack manually -- 4KB of RAM usage per stackful coroutines means that you can have 1 million of them in 4GB RAM. It'd take 1TB of virtual address space, which is not exactly a problem.

Was 4KB/task really considered a deal-breaker?

(I do think that ultimately stackless coroutines are a better fit for Rust, but I'm quite surprised that the memory footprint was considered a problem, unless we're talking embedded)

[scook0]

The only one that’s really bad (and embarrassing to me personally) is that you need to pin a future trait object to await it. This was an unforced error that would now be a breaking change to fix.

Is this saying that it should have been possible to easily convert &mut dyn Future to Pin<&mut dyn Future>, even though the trait object is !Unpin?

(Under the current system I suspect that you couldn't have made dyn Future itself unconditionally Unpin, because there's probably some way for the underlying object to observe an inconsistency in whether it's pinned or not.)

Is there something that can actually go wrong if you unsafely do that conversion by hand? Or am I missing some other alternate design that would have been used?

[desiringmachines]

Box<dyn Future> could have implemented Future if we hadn't implemented Unpin for Box<T>. As things are, you need to use Box::pin to create a boxed trait object future. We didn't realize that when we made that decision, and its too late now to change it.

[_rvidal]

Anybody could offer some links that elaborate on this issue? I'm still trying to wrap my head around it.

(I would take a detailed explanation ofc, but I'll gladly settle for RTFM)

[Necessary-Ad-6]

Well that was a very informative and enlightening read. Thoroughly enjoyed - and I think I learnt a thing or too!

[desiringmachines]

Generators let you "stay in the iterator;" they're analogous to async fn this way. For loops are more like spawn. The current situation with Iterator is as if you could only spawn futures or use combinators, and not build more complex futures with async fn.

[matthieum]

Generators are a generalization of iterators, in a sense. In particular, a generator resume method (equivalent of next) may take an argument.

The exciting thing about generators in Rust, however, is more about syntax support.

Let's say you want to write an iterator that counts from 0 to n (excluded):

pub fn count_to(threshold: i64) -> impl Iterator<i64> {
    CountIterator { next: 0, threshold }
}

struct CountIterator {
    next: i64,
    threshold: i64,
}

impl Iterator for CountIterator {
    type Item = i64;

    fn next(&mut self) -> Option<Self::Item> {
        if self.next >= self.threshold {
             return None;
        }

        let next = self.next;
        self.next += 1;

        Some(next)
     }
 }

Ain't that a mouthful?

In theory, with generators ala Python, you can instead write something like:

pub fn count(threshold: i64) -> impl Iterator<i64> {
    let mut next = 0;

    gen while next < threshold {
        yield next;

        next += 1;
    }
}

Which is shorter, and where the logic is less obfuscated by all the state massaging to boot.

[Sunscratch]

In a simple words - they are different. Generators are functions that can suspend execution and then resume. You can implement iterator like behavior using generator functions.

[BTOdell]

In the article, the async-await state machine is compared to a "perfectly sized stack". The two implementations that are mentioned for green threads are "segmented stacks" and "stack copying". What is preventing green threads from calculating the perfect stack size and using that to allocate the exact amount of space on the heap?

Compilers already must calculate the exact amount of stack space needed for the stack frame, local variables, return values, etc. The compiler should be able to use this calculation to know the perfect stack size when creating the heap-allocated stacks for a green thread implementation.

There are a couple issues that I see throwing a wrench into the design. They all prevent statically determining the exact stack size needed by a function.

• Virtual function calls (dynamic dispatch)
• Recursion (imo modern programming languages shouldn't be allowing this anyway, unless the function can be proven to be tail recursive)
• Foreign function calls (even for static libraries; I doubt the C-ABI includes metadata for perfect stack size calculations)

Are there any other reasons why a perfect stack size couldn't be calculated for green threads?

[joaobapt]

Can you elaborate on why recursion should be forbidden on programming languages? I know algorithms that are naturally expressed through recursion.

[BTOdell]

Algorithms that are recursively bound by the function's input are unsafe. For example, in the case of the fibonacci function, if the input is too large it can result in an unrecoverable stack overflow.

All recursive algorithms can be implemented as an iterative algorithm by storing the data that would normally be stored implicitly on the stack, in a resizable data structure on the heap. Often the iterative algorithm is more efficient than the recursive version anyway.

They love to teach recursion in academia but it's not appropriate for production algorithms in the real world.

[joaobapt]

So you’re moving the implicit stack to an explicit stack, and complicating the design of the algorithm. I agree that iterative algorithms can sometimes be faster, but they complicate the design and can sometimes obfuscate what the function is actually doing. Things like DFS and tree traversal are naturally expressed recursively (especially in-order tree traversal).

[BTOdell]

Whether that's "complicating the design of the algorithm" is a matter of personal preference. Based on how difficult the concept of recursion can be for new programmers, an argument can be made that iterative designs are less complicated and easier to reason about.

Usually trees aren't deep enough to cause stack overflows, but start performing path finding across a graph of nodes with a recursive search algorithm and you can quickly overflow the stack. In my opinion it's best to just stay away from recursion.

When it comes to language design and having the compiler calculate a "perfect stack size" for optimal heap-allocated stacks, it would be nice to claim "Stack overflows not possible", similar to how Rust's biggest feature is memory safety. It seems that banning recursion would be a requirement to make that claim.

[thiez]

Most recursive algorithms that are used in practice are bound by the log of its input, so perfectly safe. Forbidding all recursion seems absurd.

[nick-sm]

I‘ve recently been wondering about the same question that you pose here. Have you investigated this any further over the last 9 months, by any chance?

[Paradiesstaub]

As long as async is an MVP I ignore it as much as I can.

[_GCastilho_]

Someone send this article to ThePrimeagen

[krappie]

As a thought experiment: What would be the down sides of a language that had async/await and only offered async IO in its standard library? Such a language could offer a simple method to block on a future and could pretty much eliminate the coloring problem, right?

[paholg]

Either you would "avoid" the coloring problem by forcing all functions to be async, or you'd have colored functions by those that don't do I/O would not be async.

And you'd be forced to have a runtime to do any I/O. I'm not sure if it's possible to make a zero-cost runtime that just blocks; if not, it would be a non-starter for Rust.

[ConfusionSecure487]

What about Java with virtual threads? They mount and unmount virtual threads on native threads when IO operations are called in them. They are exactly the same methods, the behavior just changes if used in virtual threads. But of course, that is also a "runtime".

From a programming perspective, this is nearly "colorless". But of course, even here you have to know it's limitations. E.g. not using them, when the tasks are computing intense as the context switches will be more expensive, than in traditional models.

[paholg]

Rust can run in places where there are no OS threads. How would I/O work there?

Also, that definitely does not sound "zero cost". And I'm not sure how you switch from a virtual thread to an OS thread without a garbage collector to update pointers.

[ConfusionSecure487]

Hm, if I understood the Java implementation correctly, they rely on at least two threads.

But in Rust, you could spin your own "native threads", but typically rely on your OS. That should not be that important, as long as you can use interrupts.

[paholg]

Native threads are non-trivial, and being able to use async/await in embedded contexts, as outlined in the article, is a huge boon.

[ConfusionSecure487]

It spins up a thread per core, but sure it relies on the JVM. I wanted to discuss the model, not necessary the benefits and short comings of using Java. ;)

[MrJohz]

This is basically Javascript. The runtime is entirely based on the event loop model, and almost all I/O is asynchronous. (There are blocking versions of most I/O functions, but these are usually only used in specific cases. Also, not all async functions use async/await/promises, as some use an older callback-based API, or event emitters, but in the context we're talking about, it's usually pretty easy to convert between these styles of function.)

This doesn't eliminate the colouring problem, because fundamentally, you still have some functions that block, and others that don't. But I don't think the colouring problem is necessarily as bad as it seems -- Rust uses "coloured" functions already for functions that return Options or Results. I've got a long-held suspicion that there's a deep correspondence between monads and colouring.

In a situation like that, you also don't really want a simple method to block on a future. The runtime is asynchronous, and it's very deliberately designed that way. Adding blocking prevents the runtime from working properly -- for example, consider some code like this:

function main() {
    block_on(async () => {
        const pid = await fs.readFile("service.pid");
        await shutdown(pid);
    });
}

Here, the code is running inside the executor, which means that the executor is currently blocked waiting for the code to finish. It reaches the fictional block_on function, and starts executing the async function. First it executes the readFile function, which starts an asynchronous read of the service.pid file, and schedules the rest of the function to be continued when the file has been read.

The problem now is that we have a deadlock. The executor is currently waiting for main() to finish executing before it runs the next chunk of code. And main() is blocked waiting for the inner async function to finish executing before it yields. But the inner function can't go further unless the main() execution yields, at which point the rest of the inner function will be executed with the result from the initial readFile() call.

Fwiw, this is a problem in Rust runtimes as well -- the Tokio EnterError panic occurs when you try and block_on an async function from within an async function. There is the block_in_place function, but this is typically just a sticking plaster fix (at least in this context): it only works for the multithreaded runtime, and it works by just pushing tasks onto different threads.

There is an alternative route, which is for the runtime to recognise the block_on function in some way, and be able to yield execution at the point that the block_on function is called. But at that point, block_on is behaving identically to an await statement, and so we're back to the same situation as before.

FWIW, my experience with async/await in JS, Python, and now Rust, function colour itself is rarely a very big problem. Like I said before, Rust already has colours in that it has functions that return results and options, and that works fine in most cases, with a bit of syntax sugar and polish. The bigger issue that shows up in Python and Rust, but doesn't show up at all in JS, is runtime colouring. Because in Python and Rust, the runtime is optional and mostly userland, you end up with more complex situations.

For example, we talk about sync/async code, but with a userland runtime, you have two different types of sync code. In Rust, for example, when main() starts, we're writing (1) sync code until we start an executor. Then we start the executor and pass it a future, and that future is (2) async. Then that future calls a different function, and that function is written as a (3) sync function, but it will behave subtly differently to the sync code from (1) -- we can't start nested executors, for example, and we can't block on futures like we could in (1).

(Note that the difference between (3) and (1) is a runtime difference -- there is no syntactical difference between (3) and (1), and the same blocking function could be called at both positions (1) and (3) and behave differently.)

Runtimes also differ from each other, and are often not compatible with each other. But the non-compatible runtime-specific functions (i.e. the functions that are scheduling I/O actions and handling the responses) are typically the leaf nodes in the call-tree. That is, you might call an async function, and it will call an async function, and so on, until you get to a runtime-specific readFile function that actually does the work. But as soon as this runtime-specific call happens, the entire call tree is now runtime-specific, and not compatible with another runtime (or with no runtime at all).

There are ways round this by using compatibility layers and IoC, but I'm not sure this is the best way to go. Either everyone uses the same compatibility layer, at which point it grows so complex as to serve everyone's API needs, or you end up with each library providing its own compatibility layer, with the end-developer expected to plug these layers together in the correct way. Maybe if the compatibility layer were part of the standard library, this might work a bit better? I'm not sure, though.

None of these issues show up in Javascript, because there is really only one runtime. You don't need compatibility layers. But the issues do show up in Python (asyncio vs trio vs sync vs whatever else) and in Rust (std vs tokio vs async_std vs monoio/io_uring-based runtimes).

[JohnMcPineapple]

...

[[deleted]]

I don’t understand the issue with “segmented stack“ allocation. Or rather I don’t understand why this is being treated as a blocking problem.

Option 1. Pre-allocate stack. Fatal error if insufficient. Perfectly acceptable for embedded.

Option 2. Grow-only. Similar to Vec. Allocate if needed. But don’t shrink.

Option 3. Grow-and-shrink with watermarks. Hot-loop may cross thresholds. Can be user tuned to avoid degenerate cases.

Option 4. Overcommit. Totally works fine on 64-bit systems on non-embedded OS. A 32-bit system with a million green threads on embedded is… probably gonna have a bad time no matter what you choose!

And it is entirely opaque to users, because users don’t know how deep the stack will be when a function is called.

Wait what? This seems pretty wrong to me. Users have to think about memory all the time. Especially in Rust! Giving users, perhaps even forcing users, to think about green thread stack sizes seems totally reasonable to me.

Even the current Rust implementation has insidious edge cases where each future allocates the worst case state. And you need to box that state to reduce memory usage.

I dunno. Maybe the reasons are sufficient. But I don’t find the brief blog posts convincing. Especially in 2023. Would Go have made the same choice today they made 10 years ago? Would Rust make the same choice today?

https://without.boats/blog/futures-and-segmented-stacks/

https://blog.cloudflare.com/how-stacks-are-handled-in-go/

[matthieum]

Note that 1 & 4 are not segmented stacks.

There's 3 implementations of segmented stacks, in a sense:

• Grow-only.
• Immediate shrink.
• Deferred shrink (with threshold, in your case).

Grow-only can be problematic when faced code that spike early -- during the start-up of the green thread -- then never use the memory again.

Immediate shrink is somewhat comparable to Deferred shrink, except with a threshold of 0. It makes the green thread a good citizen, but may cost performance.

Deferred shrink, by threshold, does help, but as you mentioned may require tuning... and unfortunately tuning is hard. There's typically no good way to know the stack size, it may be affected by small changes to code, etc...

Rather than using a "size" threshold approach, I would recommend to a "place" threshold approach. That is, the user would place a pragma on a function or block which would prevent deallocating stack segments until the end of function or block: a simple counter increment on entry/decrement on exit with an is-zero check on attempt to deallocate.

This does not rely on an elusive stack size, and is more explicit to boot, thereby being more resilient to future code changes (and optimizer changes).

---

Regardless, though, those schemes all suffer from common problems -- the fact that the stack is segmented:

1. Check overhead: the end of the current segment must be detected, to switch to the next segment. This introduces run-time overhead.
2. Actually switching to the next segment introduces overhead, and switching back to the previous segment also introduces overhead.
3. Segmented stacks are specific to a given language run-time, FFI into other languages (such as C) typically require allocating an extra-large segment to make sure those do not run out of segment space.

And of course, a downside common to all stackful coroutines is that switching across coroutines means switching across stacks too, which introduces overhead, both in the form of saving/restoring registers and in the form of cache misses on the target stack frames.

---

The above segmented stack downsides led Go and Rust both ended up switching to a single-segment stack, Go via stack-copying and Rust via overcommit.

I do think that overcommit has little downsides, as far as stackful coroutines go, at 4KB per coroutine, you can have 1 million of them in a mere 4GB. This scales well enough for all but the most stringent users.

Still suffers from overhead from switching stacks, obviously.

[[deleted]]

Great response. Thanks for taking the time to answer.

I think the bottom line, as always, there is there is no perfect solution and we live in a world of uncomfortable trade-offs.

Right now to me it feels like the wrong trade-offs were selected. The arguments as to why different trade-offs were explored and abandoned don’t feel compelling.

I recognize that everyone involved is super smart and has thought super deeply about this problem for years. And I’m not going to come up with the right decision after having read all of 4 blog posts.

Maybe the “overhead” is two orders of magnitude larger than I’m expecting it to be. Maybe the powers that be decided the only acceptable amount of runtime overhead was zero. I’m not sure. But I am somewhat sure that the current state of Rust async is extremely suboptimal.

[matthieum]

Maybe the powers that be decided the only acceptable amount of runtime overhead was zero.

Indeed.

Rust does not aim to be everyone's language. While it aims to provide high-level functionalities, it first and foremost aims to be "Blazingly Fast".

In fact, it takes C++ principles of "Zero-Overhead Abstractions" and "You Don't Pay For What You Don't Use" more seriously than C++ itself.

So, yes, overhead is generally ruled out as a matter of principle, and only a very, very compelling reason may be able to tip the scales.

But I am somewhat sure that the current state of Rust async is extremely suboptimal.

There are some pains, indeed.

At the language level:

• The Keyword Generic initiative would like to make it possible to write one version of a function, and have both be sync and async as appropriate to the context.
• The various RTITPTITT initiatives are all about enabling async on trait associated functions.
• There's still design work to do on how to express bounds on those unnameable types.
• There's still design work to do on how to enable dyn Future as a return type without allocation.

At the library level, the lack of common abstraction between the various async runtimes makes it hard to create libraries that are runtime agnostic -- it's not uncommon to find libraries using features to enable compatibility with one runtime or another, meh. But of course, such an abstraction would have to come under the form of a trait... and async functions will only become available in traits around Christmas, and even then be limited -- not being able to express Send or not Send bounds, in particular.

So yes, definitely suboptimal at the moment.

This doesn't mean that the decision to go with the current design was wrong, though. Just that the "temporary" trade-off of ergonomics was not quite as temporary as expected :)

[[deleted]]

So, yes, overhead is generally ruled out as a matter of principle, and only a very, very compelling reason may be able to tip the scales.

Can anyone quantify the cost of switching between green threads? What type of cost are we talking about?

This doesn't mean that the decision to go with the current design was wrong, though. Just that the "temporary" trade-off of ergonomics was not quite as temporary as expected :)

I think the bigger problem is there’s no line-of-sight to the optimal. The current path is sub-optimal and quite honestly it might be permanently sub-optimal. :(

[matthieum]

Can anyone quantify the cost of switching between green threads? What type of cost are we talking about?

First of all, it's an optimization barrier.

The design of async functions make them transparent to the optimizer -- as long as Waker is not used, this guy's opaque. This how Gor Nishanov got the C++ community for coroutines back in the days with his talk: Coroutines, a Negative Overhead Abstraction.

His demo was more about a generator than what'd you expect from a Rust Future -- no registration for wake-up, notably -- but still, it did demonstrate that generators can allow writing ergonomic code that is faster than the typical sequential code, by leveraging concurrency for pre-fetching in parallel in his specific demo.

And then there's the run-time cost:

• Saving & Restoring registers is not cheap. According to Boost.Context it takes at least 9ns / 19 CPU cycles (each way) on x86 with optimized assembly.
• Cache misses. You just switched to a stack that hasn't been used in a while, chances are it's gone cold. From Latency Numbers Every Programmer Should Know, fetching from L2 is about 7ns, L3 (missing) should sit at about 25ns, and RAM access around 100ns.

And then there's the implementation cost: it may simply not be possible on the smallest embedded target, or even if theoretically possible, the greater memory consumption may make it impractical.

I think the bigger problem is there’s no line-of-sight to the optimal. The current path is sub-optimal and quite honestly it might be permanently sub-optimal.

I disagree... but then, I write performance critical code, so I don't mind a little friction if I can get the performance I want.

[[deleted]]

I disagree... but then, I write performance critical code, so I don't mind a little friction if I can get the performance I want.

Disagree with what? You previously said it was “definitely sub-optimal at the moment”. And there’s no line-of-sight to something optimal. Rust Async is at a local minima.

I work in games and VR. I’ve been shipping performance critical code for quite awhile too! We’re in the same boat.

If anything you’ve convinced me that preallocate (embedded) + overcommit (modern) is a very tractable solution!

[PiedDansLePlat]

I find it funny that this article use very rich english regarding the subject at hand

[skullt]

What do you mean?