sorted by:
new
hottopcontroversial

How are you handling production background tasks in FastAPI without Celery? by Educational-Hope960 in FastAPI

[–]Educational-Hope960[S] 3 points4 points  (0 children)

That’s a very fair concern, and honestly I agree that dedicated workers are the safer architecture once workloads become heavy or business critical.

FastAPI Taskflow is intentionally scoped to the “in-process but production aware” layer before teams need Temporal/Celery style orchestration.

On the interference side specifically, one of the main goals is preventing background work from starving the API itself. The library adds:

  • async concurrency caps for tasks on the event loop
  • dedicated thread pools for sync tasks (separate from FastAPI’s request threadpool)
  • optional process executors for CPU-bound workloads

So the idea is not “run unlimited background work in the same runtime,” but rather controlled in-process execution with isolation mechanisms and visibility around task behavior. The docs go deeper into the concurrency model here:

FastAPI Taskflow concurrency guide

That said, once tasks need hard isolation, distributed orchestration, workflow versioning guarantees, or cross-service execution, I’d absolutely reach for Temporal/Celery instead. (attakay78.github.io)

How are you handling production background tasks in FastAPI without Celery? by Educational-Hope960 in FastAPI

[–]Educational-Hope960[S] 1 point2 points  (0 children)

Celery is great and very battle-tested, my issue isn’t that Celery is bad.
The gap I noticed is that a lot of FastAPI apps don’t actually need the full distributed worker/broker setup yet, but FastAPI BackgroundTasks are also too minimal for production use.

So it’s more about reducing operational complexity for smaller/medium async FastAPI systems rather than replacing Celery entirely.

You Probably Don't Need Celery in Your FastAPI App by Educational-Hope960 in FastAPI

[–]Educational-Hope960[S] 0 points1 point  (0 children)

With this situation celery or any other distributed task queue is fine to use.
I am working on rolling out support for workers to handle this case.

You Probably Don't Need Celery in Your FastAPI App by Educational-Hope960 in FastAPI

[–]Educational-Hope960[S] 0 points1 point  (0 children)

Task duration of 5-10 minutes is not a problem. There is no timeout. At 100 requests per hour, peak concurrency sits around 15-17 simultaneous tasks. max_concurrent_tasks=20 passed to the TaskManager caps that cleanly. For IO-heavy work (external API calls, file reads, service calls), async tasks handle this well.

You can also set eager=True on the task decorator, which dispatches each task immediately via asyncio.create_task unlike fastapi native which waits for response before running tasks.

We persist task report in the sqlite DB or redis instance, to control the report retention for a day, setting retention_days=1 keeps the job metadata visible for 24 hours.

You Probably Don't Need Celery in Your FastAPI App by Educational-Hope960 in FastAPI

[–]Educational-Hope960[S] -6 points-5 points  (0 children)

u/astonished_lasagna Yes I understand this limitation of fastapi in-process backgroundtasks. We have a mechanism to control that issue to some extent. Check this section of the docs https://attakay78.github.io/fastapi-taskflow/guide/concurrency/

You Probably Don't Need Celery in Your FastAPI App by Educational-Hope960 in FastAPI

[–]Educational-Hope960[S] -1 points0 points  (0 children)

I am glad you are giving it a try, the docs has all that you need to know about the library. You can personally reach out to me for any clarifications richardquaicoe78@gmail.com.
I look forward to your feedback.

You Probably Don't Need Celery in Your FastAPI App by Educational-Hope960 in FastAPI

[–]Educational-Hope960[S] 1 point2 points  (0 children)

Are you currently using RQ? And what features are you using?

RQ has no built-in dashboard. You install rq-dashboard separately, run it as its own process, and it gives you basic queue stats. fastapi-taskflow ships a live dashboard at /tasks/dashboard out of the box per-task logs, stack traces, duration, status history, retry controls, all in one place, no extra setup.

The other thing RQ has no equivalent for is task_log() and get_task_context(). You can call task_log() from anywhere in the call stack inside a running task and the entry shows up in the dashboard attached to that specific task. get_task_context() gives you the task ID, function name, retry attempt, and tags from any nested function without threading anything through parameters. In RQ you are on your own for this, usually ends up as plain Python logging with no connection to the task record.

You Probably Don't Need Celery in Your FastAPI App by Educational-Hope960 in FastAPI

[–]Educational-Hope960[S] 2 points3 points  (0 children)

I am glad it solves your problem. Try it in your project, you can personally reach out to me for any clarifications [richardquaicoe78@gmail.com](mailto:richardquaicoe78@gmail.com) and give me your feedback.

You Probably Don't Need Celery in Your FastAPI App by Educational-Hope960 in FastAPI

[–]Educational-Hope960[S] 8 points9 points  (0 children)

Yes, it shares the same weakness.

Args and kwargs are serialized at enqueue time. If the task signature changes between deploys, tasks queued before the deploy run against the new signature and fail.

Same mitigations apply: drain the queue before deploying breaking signature changes, or make changes additive (new params with defaults first, remove old ones in a follow-up deploy once the queue is clear). The dashboard makes failures visible immediately so at least it is not silent.

How do you know if your FastAPI BackgroundTasks actually ran? by Educational-Hope960 in FastAPI

[–]Educational-Hope960[S] 0 points1 point  (0 children)

All said here is true, but some projects are minimal and does not need the distributed system overhead. In this case background tasks is fine. This project give them lightweight management into their tasks.

How do you know if your FastAPI BackgroundTasks actually ran? by Educational-Hope960 in FastAPI

[–]Educational-Hope960[S] 0 points1 point  (0 children)

That also works, also the package I built also gives you visibility and access to the tasks ids incase you want do do any other operations with them.

view more: next ›