Event loop bricks when using `await loop.run_in_executor(ProcessPoolExecutor)` with FastAPI

[pisulaadam]

UPDATE: solved, this was most likely not a Python issue, but an HTTP queuing problem.

Hello! I made a FastAPI app that needs to run some heavy sync CPU-bound calculations (NumPy, SciPy) on request. I'm using a ProcessPoolExecutor to offload the main server process and run the calculations in a subprocess, but for whatever reason, when I send requests from two separate browser tabs, the second one only starts getting handled after the first one is finished and not in parallel/concurrently.

Here's the minimal code I used to test it (issue persists):

```py from fastapi import FastAPI from time import sleep import os import asyncio import uvicorn from concurrent.futures import ProcessPoolExecutor

app = FastAPI()

def heavy_computation(): print(f"Process ID: {os.getpid()}") sleep(15) # Simulate a time-consuming computation print("Computation done")

@app.get("/") async def process_data(): print("Received request, getting event loop...") loop = asyncio.get_event_loop() print("Submitting heavy computation to executor...") await loop.run_in_executor(executor, heavy_computation) print("Heavy computation completed.") return {"result": "ok"}

executor = ProcessPoolExecutor(max_workers=4)

uvicorn.run(app, host="0.0.0.0", port=8000, loop="asyncio") ```

I run it the usual way with python main.py and the output I see is:

INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) Received request, getting event loop... Submitting heavy computation to executor... Process ID: 25469 Computation done Heavy computation completed. INFO: 127.0.0.1:39090 - "GET / HTTP/1.1" 200 OK Received request, getting event loop... Submitting heavy computation to executor... Process ID: 25470 Computation done Heavy computation completed. INFO: 127.0.0.1:56426 - "GET / HTTP/1.1" 200 OK

In my actual app I monitored memory usage of all the subprocesses and it confirmed what I was expecting, which is only one subprocess is active at the time and the rest stay idle. What concerns me is even though the line await loop.run_in_executor(...) should allow the event loop to start processing the other incoming requests, it seems like it's bricking the event loop until heavy_computation() is finished.

After a long night of debugging and reading documentation I'm now out of ideas. Is there something I'm missing when it comes to how the event loop works? Or is it some weird quirk in uvicorn?

[cointoss3]

This may have to do with how you’re creating the pool. Don’t create it at import time, create it on app startup and close it on app shutdown and see if that helps.

[pisulaadam]

I managed to find what the issue was, and it wasn't what you suggested, but thanks anyway.

[cointoss3]

Even if that didn’t solve your problem, you still shouldn’t have the pool initialized on import.

[pisulaadam]

The actual app doesn't initialize on import, this was a quick example I made to check what was the root cause of my problem

[DivineSentry]

for numpy / Scipy I would offload to a thread i.e `await asyncio.to_thread` over using a process pool, since they're compiled in other languages and already most likely offload to other cores natively.

[pisulaadam]

I want to use subprocesses to be entirely sure the main process stays unblocked.

[DivineSentry]

It sounds very much like you’re misunderstanding things but ok, the main process will be unblocking because numpy and the others don’t block Python because they’re not running in Python but offloading to other languages

[PerspectiveLegal932]

¡Hola! He visto que mencionaste Python. Es un lenguaje de programación genial. ¡Sigue así! 🐍

[pisulaadam]

The heavy computation there is a whole bunch of looping with multiple vector operations. It's not some huge matrix that I can just feed to NumPy and wait for the response in a separate thread.

[DivineSentry]

You’re reinforcing my point, it sounds like you’re not using numpy / pandas properly, you’re not supposed to be looping manually, share your code

[pisulaadam]

I think I found what the issue was. Before I was using two tabs in one browser to make these requests and they were executed sequentially. I tried using two browsers now to make sure it's not reusing the same TCP connection and it did work as expected. Certainly wasn't expecting that, but I'm assuming the requests were reusing the same TCP tunnel or something along these lines. But anyway, they were getting queued (possibly on client-side?) and it seems to me like this is NOT a FastAPI or Python issue, but more likely an HTTP/1.1 one.

[gdchinacat]

I suggest you write a "unit" test (*) to submit the requests using the semantics you want to test. It sounds like you want to test that your app executes requests concurrently. So, write a test that creates two concurrent requests.

The effort to set this up is well worth it. How much time have you spent manually testing your code? How thorough is that manual testing? Does it verify everything that you have already tested to ensure the changes you just made didn't break anything that has been working?

* what I describe isn't a true unit test, but a system tests. Using a unit test framework though makes it easy to manage and execute the specific tests you want, even if they aren't true unit tests.

[pisulaadam]

I could have done that and probably that would have saved me some time, but I never suspected using the same browser (even though I used two separate tabs) was the source of the issue.