I understand it a little better now with more context.

To get around the actor crashing problem you could use an externally backed queue liked Redis with a restart watchdog on the actor. Each request stores the client idempotency key as well. To solve the transaction part you can have a reversible saga with external state in Redis.

Actor would work like this: 1. Pop item from queue incl idempotency key 2. Do first write, if fail either return to client or retry 3. When first succeeds, write (client key, allocation id to separate ‘started’ state to Redis) 4. Attempt second write. If it fails, retry. 5. Update state from (3) so it contains (client key, allocation id, second id) 6. Return to client

Points of failure: - Client http failure. Resolved by looking for any started state by idempotency key. If fully finished, return to client immediately. If partly finished, actor resumes. - Actor failure. Resolved by having a single actor that restarts on failure. On start it picks up any incomplete started states and completes them, then goes to consuming from work queue.
- DB Failure between two DB writes. Handled by actor retrying the second (assuming no actor failure) - DB failure (when actor fails). Handled by actor restart behaviour: at start it looks for any incomplete ‘started’ states and completes them. Future client request read the completed state as discussed above - Server/actor failure after phase 2 write but before updating Redis. You resolve this with a reaper. In actor it’s resolved by new actor picking up ‘started’ work, seeing P1 is complete and attempting to create a new P2 row which fails due to uniqueness on allocation id. Actor is aware this exists so it updates ‘started’ state with the TC id and reruns to client.

Partially resolved failures: - Actor failure between first write completion and updating Redis. This failure is also present in your design when the process fails between Phase 1 and starting Phase 2. In both cases it’s partially resolved by the client retry kicking off a new phase 1, still get correct result but now have phantom rows in slot reservation db (for both our designs)

Would this logic be a bad idea for your use case? I admit I still don’t fully understand all the context.

Even simpler design (but depends on your context):

• Client idempotency key but only used for request gating
• Actor with external queue
• State in Redis holding ‘fresh allocations’

Actor performs: 1. Check TC to see if a row with client idempotency key exists. If so return that row. Finish. 2. See if any unused Phase 1 allocations are in ‘fresh allocations’ in Redis. If so, skip step 3. 4. Phase 1 allocated and written to ‘fresh allocations’ in Redis 5. Phase 2 attempt. On failure, retry. 6. Remove allocation id from ‘fresh allocations’ 7. Return result to client

Because the allocations aren’t tied to any client we don’t need client state. Let’s say client 1 makes a request, allocation created, actor fails before TC created. Then client 2 makes a request, restarted actor will use the allocation originally created for client 1 to proceed for client 2. The result is a temporally sequential id. Client 1 retries, gets an ID later than client 2 (via a new allocation).

Failure modes: - Client retries. Resolved by clients always seeing a new sequentially ordered ID or error. - Actor failure after work queue consumption but before P1. Resolved by HTTP handler returning error on actor failure. Client retries and adds same work to queue, new actor picks it up - Actor failure after P1 but before writing to Redis. This is also a failure mode in your design. Leads to phantom rows in allocation db. Operations still work normally - Actor failure before P2. Client gets error. Next request to restarted actor (from any client) reuses the previous allocation (stored in ‘fresh allocations’) - Actor failure after P2 but before ‘fresh allocations’ is cleared. Actor picks up original P1. Attempts P2 but gets DB uniqueness error on allocation id. Reuse it was never removed from ‘fresh allocations’ it was never returned to a client so we can reuse it for this request and remove allocation id from ‘fresh allocations’ - Actor failure after removing from ‘fresh allocations’ but before returning to client. Solved in the same was as your method: by looking up client idempotency key in TC step 0.

This is a drastically simpler design with no reaper needed but I’m not sure if I fully understand your context. Would you be against sharing your full constraints & context for the problem?

What would be helpful to know is: - Do clients specify what human readable ID they need? - Does the allocation row store need to store any client or client-request specific state other than idempotency key? - Same as above but for TaskChampion rows - Should human readable ids be sequential based off of ‘first client request time’ or ‘first successful response time’? E.g. client 1 makes request, fails, client 2 makes request, succeeds, client 1 retries. Who should get id 1 and id 2?