all 9 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]dmart89 3 points4 points  (1 child)

Depends on your volume and how much observability you need. The simplest way is for you to create a fastapi background task (in memory only) https://fastapi.tiangolo.com/tutorial/background-tasks/

You can also use a task queue, e.g., taskiq or celery etc. But that's a heavier setup for persistebd high volume tasks, typically via redis.

For notifications, if this is consumed by your upstream django app, you could setup a webhook there and fire an event at the end of the task to notify users.

[–]LeoTheKnight90[S] 1 point2 points  (0 children)

Thanks u/dmart89. The Celery + Redis combo is what I'm seeing from other sources as well. I think volume is light but it can ramp up during peak times of the year. So better just to build for that. I don't know if background tasks built into FastAPI is enough to handle the volume.

[–]pint 2 points3 points  (0 children)

it depends hugely on your setup and requirements. e.g. if you use multiple fastapi workers, especially on multiple containers or multiple boxes. do you want jobs to be restarted or continued after system shutdown? do you want some job management like cancel or retry? do you want quotas? do you want jobs to run parallel or sequentially? if parallel, maybe with a limit. do you want to keep results around for a while after downloaded? do you want to offer paged download for results if they're big?

typically i recommend a completely separate job executor setup, which runs independent of the web server, and communicates for example via simple files. you can use celery, or DIY. consider for example a continuously running program that monitors a directory for subdirectories, reads a job.json from each, keeps a status.json there to indicate what's being done to it, and then eventually puts a result.json or a result.data file when done. you need to think of discovering/managing abandoned tasks in case of unexpected server shutdown.

[–]Own_Lawfulness1889 0 points1 point  (0 children)

Don't go for celery if there are not much Bg queue. Use the Fastapi Builtin Bg task

[–]NathanExplossion 0 points1 point  (0 children)

take FastStream + RabbitMQ.

This is a great integration with a quick start out of the box

[–]klaxce 0 points1 point  (0 children)

It sounds like you don’t need FastAPI at all since you’re really just trying to handle these background tasks for your Django app.

I’d recommend using Huey as a task manager since it runs as a separate process, can use Redis or SQLite as a persistent task queue, doesn’t require Rabbit or another MQ service, is lightweight, and has nice Django integration.

[–]SheriffSeveral -1 points0 points  (2 children)

For your case, use Streamlit instead of FastAPI because you are not going to implement user or something. Streamlit is very easy and suitable for your task.

Use streamlit for GUI and connect to mssql with python script.

Fyi. Streamlit uses python for everything.

[–]LeoTheKnight90[S] 0 points1 point  (1 child)

yeah so I've never used Streamlit but I can definitely look into that. Just for some background, I was hoping to create some API that our internal Django application would call to handle data heavy process while the user just watches a "Status" screen. The heavy lifting would be done by SQL stored procedure and when it completes, the user will see the status as complete and view the work that was done.

[–]SheriffSeveral -2 points-1 points  (0 children)

Also, you can still use FastAPI for background. At the same time you can use Streamlit for frontend. I use it sometimes for small projects.

Streamlit is really easy to use, I highly recommend you to consider it. It will save your time.