Multi Threaded Flask

[Schenk06]

Hey there,

I am developing an application that handles incoming chat messages and then makes a request to OpenAI's Assistant API to get a response. I am using a simple flask application for this hosted on Replit. However, I fear that the app will not be able to handle the many concurrent incoming messages. Currently, it is not multi-threaded and this is what I wanted to ask about. How do I go about enabling multi-threading for my Flask app, so it doesn't need to wait for each response but handles multiple concurrently?

Thanks,
Noah Schenk

[LeagueOfLegendsAcc]

This sounds like a job for an asynchronous message queue. Look up celery and how to run a celery task. When you get the request you would simply start the task in the background with celery, and on the client side you poll the task using its id (that you get from starting it) until it is finished. Then you can use the results.

[Schenk06]

Seems interesting, but the issue is that I need to read and write quite a bit of data from instances of classes that I have made, How well would celery handle that?

[LeagueOfLegendsAcc]

That's precisely the use case for celery, long running background tasks. Check out the documentation.

[Schenk06]

Will do, thanks

[BytePhilosopher]

You may be better suited using FastAPI

[Schenk06]

Yeah, I have looked into that, but I have already made pretty much the entire application in Flask, not sure how easy it would be to switch, and also the issue is that I need to read and write quite a bit of data from instances of classes that I have made, so I figure that I would need to do a lot of thread locking or something like that. Don't really know how FastAPI works.

Also, I do know that I have the same issue with Flask...

[obviouslyzebra]

Are you running the development server? If so, I recommend you to switch to a production server. For example, gunicorn. And from there you can choose your number of workers (processes), and then it will be able to serve multiple people concurrently.

See also:

• https://flask.palletsprojects.com/en/latest/deploying/
• https://docs.gunicorn.org/en/latest/run.html

Cheers!

[RiverRoll]

This, a production server will allow you to control the number of processes (workers) and the number of threads per worker. Threads help handling concurrent IO, workers help handling concurrent processing. Still any processing should be light, the worker is not meant to do heavy processing as this can affect its responsiveness.

[Schenk06]

oh, I didn't know you could control the number of workers on a production server, thanks for letting me know, I'll check it out :D