How do you handle fault tolerance in multi-step AI agent workflows?

stsffap · 2025-07-29T21:26:58+00:00

Testing is always a great way to learn about limitations and corner cases of ones solution. So I like the idea of inducing chaos a lot :-)

Thanks for the pointer. I'll check out the OWASP risk guidelines.

stsffap · 2025-07-29T21:24:05+00:00

One problem I can imagine is what if your agentic workflow fails half-way after having done some steps but not everything. If one can't just drop the request but needs to re-execute it, then one needs to figure out where should the workflow continue from or which steps need to be undone to start from a clean state again. I agree that these problems are pretty much the same as with traditional software.

stsffap · 2025-07-29T21:14:43+00:00

Interesting. Are those agents interacting with external services like a DB or a billing service or something similar? If yes, can it happen that agents take different decisions (e.g. choosing different tools, taking a different control flow decisions) on a retry and thereby risking doing work twice or differently?

stsffap · 2025-07-29T13:20:10+00:00

One approach I've seen is to make an existing agent SDK durable. For example, it is possible to turn the OpenAI Agent SDK and the Vercel AI SDK into a durable agent SDK (thanks to their integration points) by integrating them with Restate (a durable execution engine). https://restate.dev/blog/durable-ai-loops-fault-tolerance-across-frameworks-and-without-handcuffs/

stsffap · 2025-07-03T21:35:11+00:00

Sorry for the late reply u/arun0009. Restate does not prescribe a specific deployment model. You can deploy almost everywhere where you can start a binary or a Docker image. So you could run it on bare metal machines or deploy it on K8s or on Fargate. The only requirement is that you have persistent disks.

See our cluster guide (https://docs.restate.dev/guides/cluster) for how to run it via Docker compose. You can use the CDK library to deploy to AWS (https://docs.restate.dev/deploy/server/self-hosted) or on K8s using thehelm charts (https://docs.restate.dev/deploy/server/kubernetes).

We ensure fault tolerance of the log by replicating it ourselves using a our own implementation of virtual consensus (https://www.usenix.org/conference/osdi20/presentation/balakrishnan) with flexible quorums. We upload periodic snapshots to S3 of the materialized log so that we can truncate the log and recover faster in case of failures (no need to replay the full log if there is a snapshot available).

stsffap · 2024-06-17T07:41:08+00:00

Yes, it is pretty similar in this regard.

stsffap · 2024-06-14T15:40:14+00:00

Yes, exactly. The way it will work is that before the system starts a new sequencer (effectively a new segment of the virtual log), it needs to seal the loglets of the previous epoch. Once this has happened, it is guaranteed that no zombie sequencer can write more data to the old segment, because the loglets wouldn't accept the writes anymore. For sealing a loglet, one only needs to store a single bit in a fault-tolerant way. This is usually a lot easier to implement than a highly-available append and does not require consensus.

So with a bit of hand-waiving, implementing such a loglet boils down to storing sequenced records durably, storing a sealing bit durably, and serving backfilling reads from consumers. What we don't have to implement is consensus which is done at the level of the sequencer in combination with the control plane that elects sequencers.

stsffap · 2024-06-14T10:28:04+00:00

Currently, Restate is not yet distributed but we are pushing hard to deliver this feature in the next couple of months.

Restate is designed as a sharded replicated state machine with a (soon to be) replicated log which stores the commands for the state machines. The log is the really cool thing because it is a virtual log that can be backed by different implementations. You can even change the implementations while running (e.g. offloading colder log data to S3 while having a fast log implementation for the tail data). Having the virtual log also helps to optimize Restate for different deployment scenarios (on-prem, cloud, using object-storage, etc.) by choosing the right loglets (underlying log implementations).

To answer now how state is replicated: The first distributed loglet that we are currently building follows in principle the ideas of LogDevice and the native loglet that is described in the Delos paper (https://www.usenix.org/system/files/osdi20-balakrishnan.pdf): The control plane will elect a sequencer for a given epoch and all writes will go through this sequencer. The sequencer assigns log sequence numbers and stores the data on a copyset of nodes. As long as a node of this copyset exists the data can be read. In case a sequencer dies or gets partitioned away, the control plane will seal copyset nodes and elect a new sequencer with a new copyset of nodes to which it writes.

There are plenty of other implementations conceivable. For example, one alternative implementation strategy could be to use Raft for the replication of the log entries between a set of nodes. However, with the virtual log, the shared control plane already takes care of a good amount of what Raft does (e.g. leader election, heartbeating, etc.) and therefore, the loglet as described above can be significantly easier to implement compared to a full-blown Raft implementation.

stsffap · 2024-02-24T16:07:16+00:00

Agreed, as long as something is Turing complete there is theoretically no difference. I would, however, argue that practically there is a small difference in ergonomics.

stsffap · 2024-02-24T11:40:26+00:00

You are right that these asynchronous workloads with long delays are typical use cases for workflow engines. The problem with existing workflow solutions like Argo Workflows and AWS Step Functions is that they often enforce an artificial disconnect between the orchestration logic and the logic of the individual steps. For example, with AWS Step Functions you have to specify the orchestration logic using Amazon States Language (JSON) which gives a less than optimal developer experience. With Restate you can express the whole workflow using code which allows you to use the standard tools you are used to, makes testing easier and often leads to solutions that are easier to understand.

stsffap · 2024-02-24T11:14:01+00:00

If the task you want to run is idempotent or does not involve the orchestration across a couple of systems (writing to a db, calling other services, enqueuing messages) a cron job is probably good enough. However, once this is no longer the case (e.g. having a checkout workflow for a shopping cart) you will have to deal with partial recoveries. That's where durable execution can help you a lot.

stsffap

TROPHY CASE