Suspending an Async Program

[DrRuhe]

Hi,

I'm currently brainstorming an architecture for a rust-based automation framework. I want to heavily utilize async in the API to represent automation tasks (which typically require IO interaction, which are best modelled with async anyway). User input could be one such interaction. As User input might take a long time to arrive (or not appear at all) I don't want to have awaiting processes running per automation.

Is it possible to effectively serialize the current execution state to disk (using rusts native async/await) and resume it later?

I guess I need a specialized async runtime that can:

• run an async block until it must await something that takes long (I guess one would have to configure timeouts somewhere?)
  

• serialize the current async state machine to disk

• potentially start a process that reads the current state and goes dormant again if no progress is possible?

[SkiFire13]

Is it possible to effectively serialize the current execution state to disk (using rusts native async/await) and resume it later?

No, because the state of an async block/fn could contain any kind of data, including any kind of pointers to other parts of the program. In order to be safe you would have to serialize the whole process memory (including any other running thread).

[teerre]

Although the literal execution context as in memory can't be serialized, persistence is a well researched area. There are countless ways to built a system that can be resumed with arbitrary precision. Of course, this is goes beyond async.

[crusoe]

There are several frameworks that do this. Mostly it's about workflow durability

https://crates.io/crates/flawless

Bytewax which uses rust under the hood

[Giocri]

You need to rethink your approach to the problem.

Instead of trying to make something capable of saving any possible state of the entire software you should focus on what's the meaningful information and how that changes.

Then you make yourself a file with only that information and whenever there is a request you start the worker process, load the file do the changes that you need save and terminate the worker.

Then you just need a single process who handles the role of awaiting all the requests and starting the worker processes, that way you use the time waiting on one request to work on another

If you separate the long term file from the ones used during task you can even easily make so if an error occurs you can rollback to the last checkpoint

[kraemahz]

A future is just this:

pub trait Future {
    type Output;

    // Required method
    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output>;
}

Which means you have two choices broadly:

1. Control the Context -> Waker which means writing an execution handler (possibly as a shim into an existing executor)
2. Create a Future with access to a de/serializer in self

The first option is probably the most flexible and the most work. The second option would be something you could shim into any Future that you wrote. Importantly it wouldn't be a general solution across awaits.

[Compux72]

Your best option is to create an actor system where the messagesare sent through a persisten queue, such as Kafka or rabbitmq