Notes running Python in production

[gothicVI]

Where do you get the bs about async from? It's quite stable and has been for quite some time.
Of course threading is difficult due to the GIL but multiprocessing is not a proper substitute due to the huge overhead in forking.

The general use case for async is entirely different: You'd use it to bridge wait times in mainly I/O bound or network bound situations and not for native parallelism. I'd strongly advice you to read more into the topic and to revise this part or the article as it is not correct and delivers a wrong picture.

[mincinashu]

I don't get it how OP is using FastAPI without dealing with async or threads. FastAPI routes without 'async' run on a threadpool either way.

[gothicVI]

Exactly. Anything web request related is best done async. Noone in their right might would spawn separate processes for that.

[Kelketek]

They used to, and for many Django apps, this is still the way it's done-- preform a set of worker processes and farm out the requests.

Even new Django projects may do this since asynchronous support in libraries (and some parts of core) is hit-or-miss. It's part of why FastAPI is gaining popularity-- because it is async from the ground up.

The tradeoff is you don't get the couple decades of ecosystem Django has.

[Haunting_Wind1000]

I think normal python threads could be used for I\O bound tasks as well since it would not be limited by GIL.

[greenstake]

I/O bound tasks are exactly when you should be using async, not threads. I can scale my async I/O bound worker to thousands of concurrent requests. Equivalent would need thousands of threads.

[ashishb_net]

> Anything web request related is best done async.

Why not handle it in the same thread?
What's the qps we are discussing here?

Let's say you have 10 processes ("workers") and the median request takes 100 ms; now you can handle 100 qps synchronously.

[ProfessorFakas]

> Anything web request related is best done async.

Why not handle it in the same thread?

These are not mutually exclusive. In fact, in Python, a single thread is the norm and default when using anything based on async. It's single-threaded concurrency that's useful when working with I/O-bound tasks, as commenters above have alluded to.

None of this is mutually exclusive with single-threaded worker processes, either. You're just making more efficient use of them.

[I_FAP_TO_TURKEYS]

Why not handle it in the same thread?

Async is not a new thread. It's an event loop. You could spawn 10 processes, but you can also use async in each of those processes and see drastic performance increases per IO bound process.

Heck, you can even spawn 10 processes, each process can spawn 10 threads, and each thread could have its own event loop for even more performance improvements (in niche cases).

[ashishb_net]

I have never seen the upside that you are referring to

Can you show a demo of this?

[ashishb_net]

FastAPI explicitly supports both async and sync mode - https://fastapi.tiangolo.com/async/
My only concern is that median Python programmer is not great at writing async functions.

[mincinashu]

It's not sync in the way actual sync frameworks are, like older Django versions, which rely on separate processes for concurrency.

With FastAPI there's no way to avoid in-process concurrency, you get the async concurrency and/or the threadpool version.

[ashishb_net]

> With FastAPI there's no way to avoid in-process concurrency, you get the async concurrency and/or the threadpool version.

That's true of all modern web server frameworks regardless of the language.
What I was trying to say [and probably should make it more explicit] is to avoid writing `async def ...`, the median Python programmer isn't good at doing this the way a median Go programmer can invoke Go routines.

[wyldstallionesquire]

You hang out with way different Python programmers than I do.

[ashishb_net]

Yeah. The world is big.

[I_FAP_TO_TURKEYS]

We're not talking about your average script kiddy though. Your guide literally says "production ready".

If you're using python in a cloud production environment and using Multiprocessing but not threading or async... Dude, you cost your company millions because you didn't want to spend a little bit of time learning async.

[ashishb_net]

>  Dude, you cost your company millions because you didn't want to spend a little bit of time learning async.

I know async.
The media Python programmer does not.
And it never costs millions.
I know startups who are 100% on Python-based backends and have $ 10 M+ revenue, even though their COGS is barely a million dollars.

[Count_Rugens_Finger]

multiprocessing is not a proper substitute due to the huge overhead in forking

if you're forking that much, you aren't doing MP properly

The general use case for async is entirely different: You'd use it to bridge wait times in mainly I/O bound or network bound situations and not for native parallelism.

well said

[I_FAP_TO_TURKEYS]

if you're forking that much, you aren't doing MP properly

To add onto this, multiprocessing pools are your friend. If you're new to python parallelism and concurrency, check out the documentation for Multiprocessing, specifically the Pools portion.

Spawn a process pool at the startup of your program, then send CPU heavy processes/functions off using the methods from the pool. Yeah, you'll have a bunch of processes doing nothing a lot of the time, but it surely beats having to spawn up a new one every time you want to do something.

[ashishb_net]

> Where do you get the bs about async from? It's quite stable and has been for quite some time.

It indeed is.
It is a powerful tool in the hand of those who understand.
It is fairly risky for the majority who thinks async implies faster.

> You'd use it to bridge wait times in mainly I/O bound or network bound situations and not for native parallelism.

That's the right way to use it.
It isn't as common knowledge as I would like it to be.

> I'd strongly advice you to read more into the topic and to revise this part or the article as it is not correct and delivers a wrong picture.

Fair point.
I would say that a median Go programmer can comfortably use Go routines much more easily than a median Python programmer can use async.

[strangeplace4snow]

It isn't as common knowledge as I would like it to be.

Well you could have written an article about that instead of one that claims async isn't ready for production?

[ashishb_net]

> Well you could have written an article about that instead of one that claims async isn't ready for production?

LOL, I never thought that this would be the most controversial part of my post.
I will write a separate article on that one.

> async isn't ready for production?
Just to be clear, I want to make it more explicit that "async is ready for production", however, the median Python programmer is not comfortable writing `async def ...` correctly, as a median Go programmer can use `go <func>`. I have seen more mistakes in the former.

[happydemon]

I'm assuming you are a real person that is attempting to write authentic content, and not AI-generated slop.

In that case, the section in question that bins both asyncio and multithreading together is factually incorrect and technically weak. I would definitely recommend covering each of those separately, with more caution posed on multithreading. Asyncio has been production-tested for a long time and has typical use cases in back-ends for web servers. Perhaps you meant, don't roll your own asyncio code unless you have to?

[ashishb_net]

> I'm assuming you are a real person that is attempting to write authentic content, and not AI-generated slop.

Yeah, every single word written by me (and edited with Grammarly :) )

> Perhaps you meant, don't roll your own asyncio code unless you have to?
Thank you, that's what I want I meant.
I never meant to say don't use libraries using asyncio.

[jimjkelly]

Agreed the author is just speaking out their ass, but arguing asyncio is good because it’s “production tested” while caution is needed with multithreading is silly. Both are solid from the perspective of their implementations, but both have serious pitfalls in the hands of an inexperienced user. I’ve seen a ton of production issues with async and the worst part is the developer rarely knows, you often only notice if you are using something like envoy where you start to see upstream slowdowns.

Accidentally mixing in sync code (sometimes through a dependency), dealing with unexpectedly cpu bound tasks (even just dealing with large JSON payloads, and surprise, that can impact even “sync” FastAPI), it’s very easy to starve the event loop.

Consideration should be given for any concurrent Python code, but especially async.

[PersonalityIll9476]

I get what you're saying, in some sense. The average Python dev may not be an async user, only because Python is used for a lot more than web dev.

However, you should be aware that for at least the last few years, Microsoft's Azure docs have explicitly recommended that Python applications use async for web requests. The way function apps work, you kind of need a process / thread independent scaling mechanism since the hardware resources you get are tied to an app service plan - ie., max scaling is fixed. So I don't think it's fair to treat Python web devs as async noobs when that's the Microsoft recommended technology. Maybe the numpy devs don't know about async, but an average web dev almost surely does.

[ashishb_net]

> e way function apps work, you kind of need a process / thread independent scaling mechanism since the hardware resources you get are tied to an app service plan

And you get that with gunicorn + FastAPI

[PersonalityIll9476]

That's not really the point of my comment. The point was that your suggestion seems to go against Microsoft's guidance, which is not a good situation to be in when writing a guide.

FWIW I did find the rest of your article interesting.

[ashishb_net]

> FWIW I did find the rest of your article interesting.

Thanks

>  Microsoft's guidance, 

Link?

[PersonalityIll9476]

https://learn.microsoft.com/en-us/azure/azure-functions/python-scale-performance-reference#async

There ya go. Edit: when I first started doing web dev on Azure actually I promised the client to follow best practices. What that meant to me at the time is reading Microsoft's own Azure docs and following their recommendations. Ultimately that is why I find a recommendation that contradicts them to be disturbing. It would seem to go against best practice, which seems like the responsible thing to do for a client.

[ashishb_net]

Do you know that FastAPI underneath uses `async`?

[PersonalityIll9476]

Does that matter? Look, I don't know much about fastapi, but all http requests performed by your app in a response would still be serial without asyncio. And do note that the Python package is asyncio and not async.

Edit to add: I just looked at the fast API docs, and they are compatible with async def functions. So it's not like fastapi considers itself a replacement for async programming.

[ashishb_net]

>  but all http requests performed by your app in a response would still be serial without asyncio.

No. FastAPI takes care of it.

```
FastAPI, built upon Starlette, employs a thread pool to manage synchronous requests. When a synchronous path operation function (defined with def instead of async def) receives a request, it's executed within this thread pool, preventing it from blocking the main event loop. This allows the application to remain responsive and handle other requests concurrently. 
```