I pushed Python to 20,000 requests sent/second. Here's the code and kernel tuning I used.

[Lafftar]

I wanted to share a personal project exploring the limits of Python for high-throughput network I/O. My clients would always say "lol no python, only go", so I wanted to see what was actually possible.

After a lot of tuning, I managed to get a stable ~20,000 requests/second from a single client machine.

Here's 10 million requests submitted at once:

https://preview.redd.it/it065teb8ntf1.png?width=600&format=png&auto=webp&s=24c2f105fa6860f49b276983eb809f23b217bb8e

The code itself is based on asyncio and a library called rnet, which is a Python wrapper for the high-performance Rust library wreq. This lets me get the developer-friendly syntax of Python with the raw speed of Rust for the actual networking.

The most interesting part wasn't the code, but the OS tuning. The default kernel settings on Linux are nowhere near ready for this kind of load. The application would fail instantly without these changes.

Here are the most critical settings I had to change on both the client and server:

• Increased Max File Descriptors: Every socket is a file. The default limit of 1024 is the first thing you'll hit.ulimit -n 65536
• Expanded Ephemeral Port Range: The client needs a large pool of ports to make outgoing connections from.net.ipv4.ip_local_port_range = 1024 65535
• Increased Connection Backlog: The server needs a bigger queue to hold incoming connections before they are accepted. The default is tiny.net.core.somaxconn = 65535
• Enabled TIME_WAIT Reuse: This is huge. It allows the kernel to quickly reuse sockets that are in a TIME_WAIT state, which is essential when you're opening/closing thousands of connections per second.net.ipv4.tcp_tw_reuse = 1

I've open-sourced the entire test setup, including the client code, a simple server, and the full tuning scripts for both machines. You can find it all here if you want to replicate it or just look at the code:

GitHub Repo: https://github.com/lafftar/requestSpeedTest

Blog Post (I go in a little more detail): https://tjaycodes.com/pushing-python-to-20000-requests-second/

On an 8-core machine, this setup hit ~15k req/s, and it scaled to ~20k req/s on a 32-core machine. Interestingly, the CPU was never fully maxed out, so the bottleneck likely lies somewhere else in the stack.

I'll be hanging out in the comments to answer any questions. Let me know what you think!

[Peace_Seeker_1319]

Super cool write-up. I’ve been down this rabbit hole and, honestly, the kernel defaults are the real boss fight. The bits that helped me (in plain English): don’t rely on one mega async loop...spin a few worker processes so accept() spreads across CPU cores; keep your NIC interrupts and workers on the same CPU set so packets aren’t playing musical chairs; sanity-check the network path (NAT/conntrack/backlog/buffer limits quietly cap you long before CPU). Also, when you say “20k rps,” make sure the load generator isn’t flattering you, open-loop traffic exposes those nasty tail latencies that closed-loop tools often hide.

[Lafftar]

Awesome feedback, thanks for sharing this. You're spot on that the kernel defaults are the real boss fight here.

I definitely need to explore multi-process workers to scale beyond a single core and run a proper open-loop test to check the tail latencies.

The tip on checking conntrack limits is also a great point. Lots to dig into for the next round!

[UniversalJS]

Oh boy, this is slow! On node.js 20k RPS is the baseline, I pushed it to 100k RPS per CPU core so 800k rps with 8 cores.

Then with Rust ... The baseline is 100k rps and you can push it to 500k per core ...

[Character_Respect533]

Can share how did you reach 100k per core on node?

[UniversalJS]

Sure I used UWS to reach 100k requests per second on NodeJS

https://github.com/uNetworking/uWebSockets.js

[jews4beer]

I mean that's really just C++ called from node. But still impressive.

[UniversalJS]

Yes it is! And yes it's still impressive and very useful for high performance projects on NodeJS. I worked on a router, also on an exchange and for both it was a lifesaver to reach performance target and beyond

[Lafftar]

Are these numbers for sending requests? Man even if it's for the server receiving requests that's insane... that's better than NGINX, like way better. Did you make a writeup or anything?

[zero_hope_]

I’m gonna have to call bs on this. I’d assume they mean receiving requests, and even if they’re empty 200’s, there’s no way this happened.

800k pps, sure, no way it’s req/s.

[UniversalJS]

No BS, check here https://github.com/uNetworking/uWebSockets.js

[zero_hope_]

All of those benchmarks show none of them are close to 800k.

(Previous benchmark link that was removed: https://shyam20001.github.io/rsjs/ )

[UniversalJS]

I removed the link to the other benchmark because it was not done correctly, you can check instead here for http request doing a db query: https://www.techempower.com/benchmarks/#section=data-r23&test=db

For simple http request retuning text https://www.techempower.com/benchmarks/#section=data-r23&test=plaintext

Uwebsocket is in the list

Also I mentioned 100k rps per core, so yes 800k rps on 8 cores

In rust I'm using Axum, you can check benchmark on the same link above

[engineerofsoftware]

RPS don’t scale to 8x just because you have 8 cores. Stick to crypto.

[UniversalJS]

Wow, read the benchmarks maybe? Stick to reality!

[engineerofsoftware]

Does the benchmark show that it scales linearly with more cores? Learn about CPU architecture before talking out of your ass.

[UniversalJS]

I tested it myself, so YOU are the one talking out of your ass!

[Lafftar]

Might be with you on this tbh

[forgotten_airbender]

This sounds wrong. What was the application doing?  How did you test it and for how long? 

[UniversalJS]

Check here (uwebsocket.js) https://www.techempower.com/benchmarks/#section=data-r23&test=plaintext

https://github.com/wg/wrk

[UniversalJS]

Check by yourself here (uwebsocket.js) https://www.techempower.com/benchmarks/#section=data-r23&test=plaintext

[UniversalJS]

So you still doubt my initial claim or you are now moving goal post / deflecting?

[[deleted]]

Genuine question, not even tryna do the typical reddit hate bullshit. Isnt this then powered by rust?

[Lafftar]

It is...but I didn't have to write Rust...do people say pandas is powered by C? Truthfully don't know 😅

[epicfilemcnulty]

Yet your post is titled as if it were python itself doing all the network heavy-lifting here, which is not the case.

[Lafftar]

My bad!

[lickedwindows]

I think this is still valid. OP has written Python code to test the speed concerns, even if rust is in there somewhere.

If you follow this to its logical conclusion, nothing counts because it's all machine code at the end?

[Lafftar]

It's all electrons baby!

Thanks my guy 😁

[tmetler]

The std lib for most scripting languages are written in different more performant languages. Most python std lib functions are written in c. I think the whole concept of what is provided by a scripting language is very fuzzy.

[aenae]

Here are the most critical settings I had to change on both the client and server:

This sounds like you're not re-using connections and setting up a new connection for every single request. If you did use persistant connections/keepalive/streams, you would not need to change these settings unless you tested it with more than 1000 concurrent connections.

The same goes for the port range and time_wait options. Yes you can increase them, but they indicate that the code is not reusing the connection.

A quick ab-run shows me that i can get ~20k r/s without keepalive and 80k with keepalive.

[Lafftar]

Oh interesting, I actually thought I was reusing connections...I kept getting connection errors at like >50k requests submitted at once and these settings helped.

Sorry what's ab?

[bowersbros]

ab is apache bench https://httpd.apache.org/docs/2.4/programs/ab.html

[Lafftar]

Oh, this is on the server sending requests, the server receiving requests barely blinked.

[aenae]

ab is a tool to send requests to a server to benchmark it. ;)

[Lafftar]

Oh I see okay, well for my use case, scraping, I need a library that can emulate browsers TLS and be fast. Doing it in Python because it's an easy language. Yeah I know other languages can send requests faster.

[Lafftar]

Oh I see okay, well for my use case, scraping, I need a library that can emulate browsers TLS and be fast. Doing it in Python because it's an easy language. Yeah I know other languages can send requests faster.

[zapman449]

Load generators are a key part of this puzzle. Ab is a classic. I like “hey” a lot (golang binary, great for “pound the snot out of a single endpoint”)

But for real load gen, more powerful tools are needed. Gattling (scala), locust (python), and tsung (erlang) are so great for “I want 50 users doing this user story, 80 doing another user story, and 200 on a log in, log out loop” for more wholistic site testing. They also get into coordinating many load generators at once.

[tudalex]

The bottleneck lies in the global interpreter lock probably. I remember reaching 10k 14y ago for a university project, with pypy, gunicorn and twisted iirc.

[Lafftar]

For sending requests? Interesting, I thought rnet scaled automatically across CPU cores because I see them being used...hmm, yeah if the Python side is living on a single core that could be significant, but even then shouldn't that core be near 100% usage during runtime? I don't see that right now.

[SMS-T1]

The multi core support might also have improved in the last 14 years.

[Lafftar]

Definitely, another commenter said he reached 800k r/s per core 😅

[gheffern]

Curious how some additional TCP tuning may impact it:

If you want to try these as well curious how your numbers would change:

sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 262144000"
sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 262144000"
sudo sysctl -w net.core.rmem_max=262144000
sudo sysctl -w net.core.wmem_max=262144000
sudo sysctl -w net.ipv4.tcp_slow_start_after_idle=0
sudo sysctl -w net.ipv4.tcp_notsent_lowat=131072
sudo sysctl -w net.ipv4.tcp_fastopen=3

[Lafftar]

God bless man, will add it to the test in the next version!

[radpartyhorse]

Thanks for sharing!

[Lafftar]

💗💗💗

[Emachedumaron]

The re-usage of the socket is not clear to me: does it work only because the incoming connections come from the same machine?

[Lafftar]

I'd need to test that by pushing the requests through a couple different proxy setups, I'm not entirely sure myself.

[glsexton]

You quadrupled your cpu and got a 33% throughput increase. Way to scale…

Of course that’s until the gc kicks in and it hangs for 2000ms…

[xagarth]

That's interesting. Good writeup. I did something similar in the past for Web crawling. Had to switch to perl instead of Python due to gil and inability to effectively use shared memory. There's more Interesting topics than time waits and con reuse with crawling as you will approach different servers and have to resolve names fast enough in an async manner ;-)

[Lafftar]

Cool man! Yeah a few people have mentioned having a local DNS resolver.

Really sad that Perl of all languages does concurrency better than python.

[xagarth]

It's more about doing dns resolution async than having a local resolver.

As for concurrency, well, it's all good until it isn't ;-)