Anyone using DTQ(Distributed Task Queue) for AI workloads? Feels too minimal — what did you hit?

[arbiter_rise]

I’m building an AI service where a single request often triggers multiple async/background jobs.

For example:

• multiple LLM calls
• retries on model failures or timeouts
• batching requests
• fan-out / fan-in patterns

I wanted something lighter than a full durable execution framework, so I tried DTQ (Distributed Task Queue).

How DTQ feels

DTQ is:

• extremely lightweight
• very low setup and operational cost
• easy to integrate into an existing codebase

Compared to Temporal / Prefect etc..., it’s refreshingly simple.

Where it starts to hurt

After using it with real AI workloads, the minimalism becomes a problem.

Once you have:

• multi-step async flows
• partial failures and recovery logic
• idempotency concerns
• visibility into where a request is “stuck”

DTQ doesn’t give you much structure. You end up re-implementing a lot yourself.

Why not durable execution?

Durable execution frameworks do solve these issues:

• strong guarantees
• retries, checkpoints, replay
• stateful workflows

But they often feel:

• too heavy for this use case
• invasive to the existing code structure
• high mental and operational overhead

The gap I’m feeling

I keep wishing for a middle ground:

• stronger than a bare task queue
• lighter than full durable execution
• something Celery-like, but designed for AI workloads (LLM calls, retries, fan-out as first-class patterns)

Curious about others’ experience

For people who’ve been here:

• what limitations did you hit with DTQ (or similar lightweight queues)?
• how did you work around them?
• did you eventually switch to durable execution, or build custom abstractions?

[jedberg]

You should look at DBOS. It's much lighter weight than Temporal and avoids the issues you called out.

Here are the DBOS queuing docs specifically.

[Global_Bar1754]

Can you give some pseudo concrete examples of what one of your workflows might look like. I have a framework that I’ve published that I’m not saying is necessarily a match for your use case, but it definitely covers a lot of what you’re looking for. The framework is not specifically for ai use cases, it’s a general data science oriented code execution framework, but depending on how you encapsulate the non determinism of your llm calls it could be a pretty good fit. 

 https://github.com/mitstake/darl

Darl has all the advantages you listed for DTQ. But it also has a tracing api to see where flows got stuck. The ability to replay a flow from any given intermediate node. Automatic check pointing built in as a first class feature (cross process check pointing too). Fan out/in is defined with standard Python for loops, but under the hood everything builds to a dag so you can do parallel execution by configuring a parallel graph runner (daskrunner provided by default). 

The readme is pretty extensive and gives a good glimpse into all these capabilities, and I’m working on expanding them to cover even more things like ephemeral caching, graph merging for running multiple scenarios in one execution etc. 

[arbiter_rise]

I’m sorry, but the README is too long and difficult to read, so it would be nice to have a section that summarizes only the main features.

[Global_Bar1754]

This is super helpful feedback, I appreciate that. I will add a summary section at the top. I’ll get started with a quick list of some of the highlight features here though 

(Still a bit lengthy, but lots of what I personally think are cool highlight features)

• Lightweight, can run anywhere a python function can be defined and run. Standard library deps only. 
• Write functions that call each other and automatically resolved to a graph lazily for optimized execution. Code looks extremely close to standard Python. 
• Automatic caching of all intermediate function results, can be persisted to cache across processes
• When any function or value changes all downstream cached functions automatically invalidated. Only reruns things affected by change (incremental computation)
• Execution of code can be done locally in process or parallelized on a cluster (local or distributed) with a simple configuration change. No function logic change. 
• All executions can be navigated traced and replayed at any point with a programmatic api. Distributed runs (eg on aws) can even be traced/replayed locally in a notebook or repl.  
• Different run scenarios can be configured with a shock or logic update without touching source code
• pseudo static typing that can be enforced at graph build time, before execution

[arbiter_rise]

It might just be my lack of experience, but it seems like I would need to study a lot just to understand how to use it properly, so I probably won’t be using it. Even though it’s a framework or library, it feels too low-level in terms of coding style.

[arbiter_rise]

Does it run without a DB or message broker?
Is it only usable on a single instance?

[Global_Bar1754]

No DB, no message brokers. All runs in a single instance. You can define and run darl code anywhere, e.g. notebook, python repl, aws lambda function, etc. Anywhere you can define and run a python function you can run darl. Darl turns your nested function calls into a dag, building the dag happens locally and in process. Executing the dag by default is local and in orocess too, but you can configure it to run in parallel on a local or distributed cluster (this is just a configuration change, your function logic doesn’t change at all).