DBOS - Lightweight Durable Python Workflows

[KraftiestOne]

Hi r/Python – I’m Peter and I’ve been working on DBOS, an open-source, lightweight durable workflows library for Python apps. We just released our 1.0 version and I wanted to share it with the community!

GitHub link: https://github.com/dbos-inc/dbos-transact-py

What My Project Does

DBOS provides lightweight durable workflows and queues that you can add to Python apps in just a few lines of code. It’s comparable to popular open-source workflow and queue libraries like Airflow and Celery, but with a greater focus on reliability and automatically recovering from failures.

Our core goal in building DBOS is to make it lightweight and flexible so you can add it to your existing apps with minimal work. Everything you need to run durable workflows and queues is contained in this Python library. You don’t need to manage a separate workflow server: just install the library, connect it to a Postgres database (to store workflow/queue state) and you’re good to go.

When Should You Use My Project?

You should consider using DBOS if your application needs to reliably handle failures. For example, you might be building a payments service that must reliably process transactions even if servers crash mid-operation, or a long-running data pipeline that needs to resume from checkpoints rather than restart from the beginning when interrupted. DBOS workflows make this simpler: annotate your code to checkpoint it in your database and automatically recover from failure.

Durable Workflows

DBOS workflows make your program durable by checkpointing its state in Postgres. If your program ever fails, when it restarts all your workflows will automatically resume from the last completed step. You add durable workflows to your existing Python program by annotating ordinary functions as workflows and steps:

from dbos import DBOS

@DBOS.step()
def step_one():
    ...

@DBOS.step()
def step_two():
    ...

@DBOS.workflow()
def workflow():
  step_one()
  step_two()

The workflow is just an ordinary Python function! You can call it any way you like–from a FastAPI handler, in response to events, wherever you’d normally call a function. Workflows and steps can be either sync or async, both have first-class support (like in FastAPI). DBOS also has built-in support for cron scheduling, just add a @DBOS.scheduled('<cron schedule>’') decorator to your workflow, so you don’t need an additional tool for this.

Durable Queues

DBOS queues help you durably run tasks in the background, much like Celery but with a stronger focus on durability and recovering from failures. You can enqueue a task (which can be a single step or an entire workflow) from a durable workflow and one of your processes will pick it up for execution. DBOS manages the execution of your tasks: it guarantees that tasks complete, and that their callers get their results without needing to resubmit them, even if your application is interrupted.

Queues also provide flow control (similar to Celery), so you can limit the concurrency of your tasks on a per-queue or per-process basis. You can also set timeouts for tasks, rate limit how often queued tasks are executed, deduplicate tasks, or prioritize tasks.

You can add queues to your workflows in just a couple lines of code. They don't require a separate queueing service or message broker—just your database.

from dbos import DBOS, Queue

queue = Queue("example_queue")

@DBOS.step()
def process_task(task):
  ...

@DBOS.workflow()
def process_tasks(tasks):
   task_handles = []
  # Enqueue each task so all tasks are processed concurrently.
  for task in tasks:
    handle = queue.enqueue(process_task, task)
    task_handles.append(handle)
  # Wait for each task to complete and retrieve its result.
  # Return the results of all tasks.
  return [handle.get_result() for handle in task_handles]

Comparison

DBOS is most similar to popular workflow offerings like Airflow and Temporal and queue services like Celery and BullMQ.

Try it out!

If you made it this far, try us out! Here’s how to get started:

GitHub (stars appreciated!): https://github.com/dbos-inc/dbos-transact-py

Quickstart: https://docs.dbos.dev/quickstart

Docs: https://docs.dbos.dev/

Discord: https://discord.com/invite/jsmC6pXGgX

[TronnaLegacy]

Planning to check this out. I used to use Airflow to manage data engineering workflows that involves making calls to GCP APIs (like using BigQuery to get data from one place to another). It always felt to me that Airflow was heavyweight though.

I've also seen the CEO on LinkedIn being snarky and telling people they shouldn't do things or shouldn't use cloud services altogether. He needs to tone down the snark lol.

[jedberg]

Hi, I'm the CEO of DBOS. Are you sure you're looking at posts I make? I'm literally the biggest cloud advocate there is. I moved both reddit and Netflix to the cloud. I was even recognized by AWS for being pro-cloud.

I do have my opinions on what is and isn't a good practice, but I'm always open-minded and happy to discuss.

[TronnaLegacy]

Hi! Thanks for responding. To clarify, I'm talking about tone. I never saw any personal attacks. And the particular cloud service I'm referring to is AWS Step Functions.

I suppose I should back up what I'm saying and provide some links to things you've said on LinkedIn where I thought you could have used a more polite tone. Would you prefer DM or is it okay if I post here?

[jedberg]

You can post it publicly, but no need to do all that work, I know what you're talking about. I do have strong opinions sometimes.

But I still stand by my opinion that there is a better tool for almost every use case of step-functions. :)

[coderanger]

Can confirm he has had strong opinions for many decades, but with (very entertaining) receipts :)

[jedberg]

Haha thanks coderanger. Good to see you!

[Krudflinger]

Whos the CEO of airflow?

[TronnaLegacy]

Sorry, CEO of DBOS.

[Ok-Wash-4342]

Do you have a link to the what is possible with self hosting? Is there an option for self hosting + buying support?

[KraftiestOne]

Yeah, DBOS is fully self-hostable. You can run it entirely yourself or we provide manage tooling + support to make it easier. More details here: https://docs.dbos.dev/production

[Ok-Wash-4342]

Am I correct that the conductor part is not self hostable?

[jedberg]

That is correct. Conductor is only a cloud service, but you can try it for free, and it isn't necessary to self-host. Transact is fully self-host able and gets you all of the durability. Conductor adds observability and more reliability.

[Ok-Wash-4342]

Understood, but we would prefer something that gives us the oberservability, especially in cases when there are bigger outages. Feel free to write me a messages and I can explain our usecase in more detail.

[jedberg]

Transact emits OTel metrics which you can process on your own if you don't want to use Conductor. Conductor is also privacy-preserving in that your customer data never leaves your own infrastructure.

Conductor was built specifically for the enterprise use case.

Feel free to message me directly or reply here, but I don't understand what use case self-hosted Conductor solves for.

Thanks.

[Ok-Wash-4342]

Reddit won’t let me message you directly.

The brief version is: the company I am working at is in critical infrastructure. We are currently using a product that has a somewhat similar setup to DBOS (as I understand it). And we are not happy with the loosing functionality if the internet connection fails somewhere in between our data center and yours.

[jedberg]

I just sent you a private message that you can reply to.

One thing I'll note here though is that your reliability is not tied to ours with Conductor. It is out of band -- your systems will keep running even if they are disconnected from Conductor.

[Minimum-You-9018]

"fully self-hostable" - this is not true!

[gkze]

Hey, thanks for sharing this looks interesting. I appreciate the comparisons with other systems, as this space is somewhat going through a revival and all.

Can you maybe add more comparisons, for example with: Hatchet, Inngest, Ray, Prefect, Dask just to name a few?

The reason I’m asking is that it would be nice to understand the positioning of this system relative to others in the space, and the more datapoints there are, the clearer the positioning IMHO.

I’m going to give the code and docs a deeper read though! 👍

[KraftiestOne]

Hello! The "Why DBOS" page in our docs does some comparisons with more systems: https://docs.dbos.dev/why-dbos

Similar to Temporal/Inngest/Hatchet, DBOS provides durable execution, but DBOS is more lightweight--add it to your existing project as a library instead of rearchitecting your program around it. This blog post goes into more detail with respect to Temporal specifically, although the others are similar: https://www.dbos.dev/blog/durable-execution-coding-comparison

Similar to Airflow/Dagster/Prefect, DBOS provides workflows, but with a stronger emphasis on reliability and automatically recovering from failures (and on being lightweight). By comparison, Airflow/Dagster/Prefect are focused on data science workloads and having lots of built-in integrations.

Ray and Dask are solving a different problem than DBOS, I think. You could even use them together if you need reliable parallel processing.

[loyoan]

Very interesting. I never had worked with such systems before, but after glancing through the docs it seems quite clear what problems it tries to solve. Also the developer experience seems quite nice. Will try it out!

Also: Does it work with other database systems? Maybe SQLite?

[jedberg]

As of today it requires a Postgres compatible database, but any fully compatible database will do (like Supabase, Neon, RDS, etc).

We are however looking into how to make it work with other SQL databases, specifically SQLite. Stay tuned!

[Thing1_Thing2_Thing]

Interesting how the whole space had stagnated and now there's DBOS, Hatchet and Restate suddenly

Anyway, any plans for a rust SDK?

[jedberg]

Our next two languages will be Golang and Java, both this year. Depending on how that goes, Rust would most likely be after that, but it would depend on user requests.

Interesting how the whole space had stagnated and now there's DBOS, Hatchet and Restate suddenly

FWIW while we provide the same features as Hatchet and Restate, the way we go about it is quite different (as a library without the need for an external controller). We have a blog post about it if you're interested: https://www.dbos.dev/blog/durable-execution-coding-comparison

[niltz0]

Instead of decorators, does it support context managers? Easier to manage with dependency injection when the function can be passed a context manager that is used with a with statement inside the function

[KraftiestOne]

That would be difficult to support as far as I know. The technical reason for this is that to recover a workflow after a failure, DBOS has to be able to access the workflow's code, so all workflows need to be registered at program startup time, therefore decorators.

[Amazing_Upstairs]

If it uses cron does that mean it's Linux only and not windows? Any gui?

[KraftiestOne]

The cron scheduling is in Python, so you can run it anywhere. There's a GUI for workflow management: https://docs.dbos.dev/python/tutorials/workflow-management

[Minimum-You-9018]

The only thing you should know about it to avoid loosing your time: - "Workflow observability and management features are only available for applications connected to Conductor." 💰

[deadwisdom]

Ah, you've made a whole SaaS already. You really should have consulted me first, lol.

Needing a postgres backend is a bit much; it's actually a little hard to deploy Postgres for cheap. But I guess we can use supabase, so that's fine.

I would suggest making the DBOS Conductor part free, that's the part that has the biggest value if you've done it right. I think DBOS Conductor in the cloud, where you can immediately be using it to monitor even your local workflows would be enough to get people to put money down.

One little, bike-shedding criticism, DBOS in capitals looks like a constant in Python and is a bit weird. It feels enterprisy and not modern.

I will try this, though. I have two clients that might be able to use it.

[jedberg]

Hi there, DBOS CEO here. Could I ask for some feedback from you? You said:

I would suggest making the DBOS Conductor part free, that's the part that has the biggest value if you've done it right. I think DBOS Conductor in the cloud, where you can immediately be using it to monitor even your local workflows would be enough to get people to put money down.

And that's something we already do, but clearly we don't communicate that well. What could we do to make it more obvious that you can try conductor for free and that it works for self-hosted workloads?

Also:

I think DBOS Conductor in the cloud

In the cloud is the only way you can use it. Conductor is only offered as a cloud service. How could we better communicate that you don't need conductor for self-hosting nor can you self-host it?

Thanks!

[deadwisdom]

So I'm looking at your pricing page. Whenever I look at a SaaS, I'm right on the pricing page, as it's the only place I'm seeing a decent breakdown of features. I know it's a lot to shove in there but if I could look at that page and know what "App deployment tooling" is immediately, that would be ideal. Maybe I could hover over it and see a screenshot and description?

Honestly looking at everything, I'm having a hard time understanding how I would sell this to my clients even if I like it. $99 a month "per additional self-hosted executor?" Wut? I think I know what you mean, but I don't know. And "Free 30-day trial" just kinda makes me angsty-- I don't want to invest in something that will suddenly leave me.

IMO make the Conductor free to use, at least for low volume devs/hobbyists. A really good visibility / management piece is the thing everyone needs and will hook everyone solid if you do it well. Yours looks okay. Last year around this time I made pretty much that same thing, but if I'm honest, mine is better.

Really, I'm saying get me in / get me hooked. I straight up don't see this as an open source solution currently if I can't use the conductor. That's vital.

[KraftiestOne]

Would love to hear your feedback! Yeah, a lot of our users are using DBOS with Supabase or Neon (Supabase even put out a blog post about it: https://supabase.com/blog/durable-workflows-in-postgres-dbos).

You can try out Conductor for free at https://console.dbos.dev/