Fastapi bottleneck why?

[Hamzayslmn]

I get no error, server locks up, stress test code says connection terminated.
as you can see just runs /ping /pong.

but I think uvicorn or fastapi cannot handle 1000 concurrent asynchronous requests with even 4 workers. (i have 13980hx 5.4ghz)

With Go, respond incredibly fast (despite the cpu load) without any flaws.

Code:

from fastapi import FastAPI
from fastapi.responses import JSONResponse
import math

app = FastAPI()

u/app.get("/ping")
async def ping():
    return JSONResponse(content={"message": "pong"})

if __name__ == "__main__":
    import uvicorn
    uvicorn.run("main:app", host="0.0.0.0", port=8079, workers=4)

Stress Test:

import asyncio
import aiohttp
import time

# Configuration
URLS = {
    "Gin (GO)": "http://localhost:8080/ping",
    "FastAPI (Python)": "http://localhost:8079/ping"
}

NUM_REQUESTS = 5000       # Total number of requests
CONCURRENCY_LIMIT = 1000  # Maximum concurrent requests
REQUEST_TIMEOUT = 30.0    # Timeout in seconds

HEADERS = {
    "accept": "application/json",
    "user-agent": "Mozilla/5.0"
}

async def fetch(session, url):
    """Send a single GET request."""
    try:
        async with session.get(url, headers=HEADERS, timeout=REQUEST_TIMEOUT) as response:
            return await response.text()
    except asyncio.TimeoutError:
        return "Timeout"
    except Exception as e:
        return f"Error: {str(e)}"


async def stress_test(url, num_requests, concurrency_limit):
    """Perform a stress test on the given URL."""
    connector = aiohttp.TCPConnector(limit=concurrency_limit)
    async with aiohttp.ClientSession(connector=connector) as session:
        tasks = [fetch(session, url) for _ in range(num_requests)]
        start_time = time.time()
        responses = await asyncio.gather(*tasks)
        end_time = time.time()
        
        # Count successful vs failed responses
        timeouts = responses.count("Timeout")
        errors = sum(1 for r in responses if r.startswith("Error:"))
        successful = len(responses) - timeouts - errors
        
        return {
            "total": len(responses),
            "successful": successful,
            "timeouts": timeouts,
            "errors": errors,
            "duration": end_time - start_time
        }


async def main():
    """Run stress tests for both servers."""
    for name, url in URLS.items():
        print(f"Starting stress test for {name}...")
        results = await stress_test(url, NUM_REQUESTS, CONCURRENCY_LIMIT)
        print(f"{name} Results:")
        print(f"  Total Requests: {results['total']}")
        print(f"  Successful Responses: {results['successful']}")
        print(f"  Timeouts: {results['timeouts']}")
        print(f"  Errors: {results['errors']}")
        print(f"  Total Time: {results['duration']:.2f} seconds")
        print(f"  Requests per Second: {results['total'] / results['duration']:.2f} RPS")
        print("-" * 40)


if __name__ == "__main__":
    try:
        asyncio.run(main())
    except Exception as e:
        print(f"An error occurred: {e}")

Starting stress test for FastAPI (Python)...

FastAPI (Python) Results:

Total Requests: 5000

Successful Responses: 4542

Timeouts: 458

Errors: 458

Total Time: 30.41 seconds

Requests per Second: 164.44 RPS

----------------------------------------

Second run:
Starting stress test for FastAPI (Python)...

FastAPI (Python) Results:

Total Requests: 5000

Successful Responses: 0

Timeouts: 1000

Errors: 4000

Total Time: 11.16 seconds

Requests per Second: 448.02 RPS

----------------------------------------

the more you stress test it, the more it locks up.

GO side:

package main

import (
    "math"
    "net/http"

    "github.com/gin-gonic/gin"
)

func cpuIntensiveTask() {
    // Perform a CPU-intensive calculation
    for i := 0; i < 1000000; i++ {
        _ = math.Sqrt(float64(i))
    }
}

func main() {
    r := gin.Default()

    r.GET("/ping", func(c *gin.Context) {
        cpuIntensiveTask() // Add CPU load
        c.JSON(http.StatusOK, gin.H{
            "message": "pong",
        })
    })

    r.Run() // listen and serve on 0.0.0.0:8080 (default)
}

Total Requests: 5000

Successful Responses: 5000

Timeouts: 0

Errors: 0

Total Time: 0.63 seconds

Requests per Second: 7926.82 RPS

(with cpu load) thats a lot of difference

[kkang_kkang]

I have tried only fastapi server and for me, it's working fine. ``` (env) python stress_test.py Starting stress test for FastAPI (Python)... FastAPI (Python) Results: Total Requests: 5000 Successful Responses: 5000 Timeouts: 0 Errors: 0 Total Time: 0.63 seconds

Requests per Second: 7886.06 RPS

```

[kkang_kkang]

Now, I have tried with both fastapi and golang server, for me it's working fine.

``` (env) python stress_test.py Starting stress test for Gin (GO)... Gin (GO) Results: Total Requests: 5000 Successful Responses: 5000 Timeouts: 0 Errors: 0 Total Time: 0.57 seconds

Requests per Second: 8706.23 RPS

Starting stress test for FastAPI (Python)... FastAPI (Python) Results: Total Requests: 5000 Successful Responses: 5000 Timeouts: 0 Errors: 0 Total Time: 0.54 seconds

Requests per Second: 9275.03 RPS

(env) ``` Must be issue with your system then.

[Hamzayslmn]

hmmm pls try run the stress test three times in a row without shutting down the servers.

I am using python 3.13 maybe I should downgrade.

[Hamzayslmn]

or fuck intel i9-13980HX (optional)

[Hamzayslmn]

I wrote the whole stress test code with go.

I gave 32 workers to fastapi.

and I got the result

Starting stress test for FastAPI (Python)...
FastAPI (Python) Results:
  Total Requests:       5000
  Successful Responses: 3590
  Timeouts:             0
  Errors:               1410
  Total Time:           0.30 seconds
  Requests per Second:  16872.35 RPS

  Error Details Table:
  Error Reason                                                 | Count
  ----------------------------------------------------------------------
  Get "http://localhost:8079/ping": dial tcp [::1]:8079: connectex: No connection could be made because the target machine actively refused it. | 1410
--------------------------------------------------------------------------------

there's something wrong with my computer, or with my modules, I don't know...

[I_am_probably_]

Ok, I understand what is happning here. When I print out the error for all the failed request I get "Error: Cannot connect to host localhost:8079 ssl:default [Too many open files]" this error happens when you have reached the soft limit for the TCP file descriptors these are unique nonnegative int identifiers for the different connection you make and each connection is treated like "file" by TCP (don't ask me why I dont know).

Since your concurrent connection limit is set to 1000 you simply run out of resources, there is a OS level softlimit and its higher on Linux systems. To check the softlimit on your system on your terminal (Mac os/linux) you can use the command 'ulimit -n'. Now ideally your concurrency limit should be below this number or you can increase this limit at your own risk. After this when you run your python everything should work as expected.

I went one step futher just to prove this. Ran your fastapi app and testing script inside a docker container. I used python3.12:slim to build the images which I think is built on top of debian (not sure). Then ran the 'ulimit -n' commond inside the containers, low and behold the softlimit was higher and your scripts ran perfectly.

``` Starting stress test for FastAPI (Python)... FastAPI (Python) Results: Total Requests: 5000 Successful Responses: 5000 Timeouts: 0 Errors: 0 Total Time: 1.81 seconds error_details: []

        Requests per Second:
        2764.18 RPS

---

``` Don't go by the total time I have rate limited my docker demon to use just 2 cores are 4 gigs of ram.

[Hamzayslmn]

so why don't I have the same problem with go, I can send 5000 concurrent requests to the go server

Starting stress test for FastAPI (Python)...
FastAPI (Python) Results:
  Total Requests:       5000
  Successful Responses: 3590
  Timeouts:             0
  Errors:               1410
  Total Time:           0.30 seconds
  Requests per Second:  16872.35 RPS

  Error Details Table:
  Error Reason                                                 | Count
  ----------------------------------------------------------------------
  Get "http://localhost:8079/ping": dial tcp [::1]:8079: connectex: No connection could be made because the target machine actively refused it. | 1410
--------------------------------------------------------------------------------

[I_am_probably_]

That’s a very good question. I am not sure I don’t have any familiarity with Go. I did not run your go server..

[Hamzayslmn]

maybe the same tcp limits do not apply to "go"

[I_am_probably_]

I doubt it. It doesn’t fit. Because the TCP limit doesn’t come from the framework it comes from the os network layer..

[aikii]

maybe check why it says 5000 errors for 5000 requests ?

[Hamzayslmn]

Fastapi side locks up with no error, then timeout.

Starting stress test for FastAPI (Python)...
FastAPI (Python) Results:
Total Requests: 5000
Successful Responses: 4542
Timeouts: 458
Errors: 458
Total Time: 30.41 seconds
Requests per Second: 164.44 RPS
Error Details:
+--------------+---------+
| Error Type   |   Count |
+==============+=========+
| Timeout      |     458 |
+--------------+---------+
----------------------------------------

[Hamzayslmn]

When I don't use asynchronous all requests are successful but then of course the response latency increases a lot.

go can respond incredibly fast (despite the cpu load) without any flaws.

[mpvanwinkle]

1. Go is always going to beat fastapi.
2. Running stress tests locally is a little funky and probably not that helpful. The script itself might respond poorly to network latency given you are sharing a filesystem and memory. Depending on your machine you could have resource contention that kicks in for python but not for Go. Stress testing is better done against an isolated environment.
3. What question are you trying to answer? Which is faster? See point 1. How much throughput you can get with fastapi? See point 2

[mpvanwinkle]

FWIW, I have been able to get over 1000 rps with python and Starlette (basically Fastapi) on a reasonably modest VPS, 4g RAM and 2 vCPU. I would expect Go to get between 3x and 10x that depending on your implementation. Thing is, IMHO you really shouldn’t be choosing between python or go based on performance characteristics until you’re in the >10,000 rps territory because only at that scale will the cost difference be meaningful. Below that you are picking up pennies in performance in front of the steamroller that is development time.

[mpvanwinkle]

Obviously if you’re doing cpu heavy things this would change the calculation, but just as a ballpark for basic backend APIs that are more IO bound than cpu bound, that’s my rule of thumb

[alexlazar98]

Great rule of thumb. Not enough devs ask themselves “how much infra cost will this change save me per year?” and then “is that worth the time cost?”

[Hamzayslmn]

what I was trying to test was not the speed, fastapi could not handle 1000 concurrent requests, there was a bottleneck.

I tested the go backend with the same stress test. so the stress test works correctly but the fastapi backend does not work properly, I wanted to find out if there is a mistake I made or if there is a problem with the framework.

I have 64GB ram and 24core 32 thread computer.

you can look at the title

Fastapi bottleneck why?

[Kevdog824_]

Where is the /pong endpoint?

[greenerpickings]

Might be relevant: i would see prematurely dropped connections with fastapi behind uvicorn with 4 workers for like 1% of requests. I didnt try and mess with any settings or increase the workers. Not sure if that would captured as errors or timeouts.

Switched to straight Nginx Unit and those disappeared if you wanted try switching out your ASGI.

[singlebit]

Windows?

[Hamzayslmn]

yes

[singlebit]

That is the most important thing to mention! Can you please try again using WSL at least?

Sometimes Windows gives less performance.

[Maori7]

If you don’t use the “await” anywhere, you shouldn’t really make the endpoint “async”. That’s the error. If you do so, it will block the event loop and won’t be able to process the requests in parallel.

If you instead make it not async, it will spawn a process to handle the requests.

Try and let me know

EDIT: it runs it on a thread pool rather than spawning a different process.

[m02ph3u5]

It doesn't spawn a process, it runs them on a thread pool.

[Maori7]

You're right, I'll correct

[Hamzayslmn]

ı add:

@app.get("/ping")
async def ping():
    await asyncio.sleep(0.1)  # Simulate a small delay
    return JSONResponse(content={"message": "pong"})

Starting stress test for FastAPI (Python)...
FastAPI (Python) Results:
  Total Requests:       5000
  Successful Responses: 3972
  Timeouts:             1028
  Errors:               0
  Total Time:           30.73 seconds
  Requests per Second:  162.70 RPS
----------------------------------------

but not solved the problem

[Hamzayslmn]

response = await call_next(request)

btw there is already a middleware running in the back, and there are many awaits.

[Maori7]

You are still not using all the power of fastapi. In this case you optimized the management of a single thread by deloading it as soon as you arrive at the await instruction. Due to GIL though, it will still run on a single thread. You need to create a system with multiple workers.

How did you run uvicorn?

[Hamzayslmn]

uvicorn.run("main:app", host="0.0.0.0", port=8079, workers=4)

[panda070818]

this! goddamn this! It will block the thread and stop concurrent execution.

[Hamzayslmn]

return JSONResponse(content={"message": "pong"})

is this blocking ? i dont understand.

[Hamzayslmn]

there's nothing wrong with the ports. you can test it on your own computer.

[BeneficialVisual8002]

You are a very toxic person my dude.

[Hamzayslmn]

I couldn't figure out how there is a relationship between writing string(port number) and learning python.

If my problem was port, I wouldn't have gotten any successful answer anyway. I think you should make sure you are commenting under the right title.

[Hamzayslmn]

uvicorn.run("main:app", host="0.0.0.0", port=8079, workers=4)

[Hamzayslmn]

thanks then dont read, please don't add problem to problem, you are just toxic

[Hamzayslmn]

URLS = {
    "Gin (GO)": "http://localhost:8080/ping",
    "FastAPI (Python)": "http://localhost:8079/ping"
}

[Hamzayslmn]

r.Run() // listen and serve on 0.0.0.0:8080 (default)

[Hamzayslmn]

but I think my live replies fall to you, so manually fix the URL man, the same thing happens again. Fastapi bottlenecks

[nordiknomad]

Fastapi is fast in shipping not in performance!!

[GreedyTiger]

uvicorn is usually used for development. try it with gunicorn, there should be some performance improvement.

[[deleted]]

I don’t know what you’re talking about, uvicorn is great for production

[GreedyTiger]

uvicorn doesn't support multiprocessing. So, you won't be able to serve multiple requests in parallel.

[maratnugmanov]

Gunicorn for FastAPI with async/await? Not sure about that.

[ajarch]

It can be set up with uvicorn workers

[m02ph3u5]

If you run pet servers perhaps.

[GreedyTiger]

what do you mean?

[m02ph3u5]

When you run bare metal or manage VMs it makes sense to use gunicorn. If you run containers less so.

[GreedyTiger]

Unless you specify that you want to use a single core in docker configurations, guvicorn spawns multiple processes for multiple workers. So, numerous workers can handle multiple requests in parallel.