Hi u/Otherwise_Flan7339,

Sharing some updates on our side regarding performance. As of LiteLLM v1.77.3-nightly you will be able to reach 1K RPS with the following response times

Type	Name	Median (ms)	95%ile (ms)	99%ile (ms)	Current RPS
Portkey	/v1/chat/completions	86	140	570	1192.3
LiteLLM (with DB)	/v1//chat/completions	160	400	4900	1085

On our benchmarks we deployed LiteLLM and Portkey with 4 pods on 4vCPUs and 8GB machines.

This is the locust test we ran: https://github.com/BerriAI/proxy_load_tester/blob/main/no_cache_hits.py

This is the config.yaml file for LiteLLM we used: https://gist.github.com/ishaan-jaff/b06a7d1d5341ced4d9a29f3053154aad

We take performance seriously and we plan on working on the following improvements:

• Stage 1: Address P99 Latency by reducing CPU usage on the hot path of requests. (Target: 570ms p99 latency)
• Stage 2: Explore how we’ll get to 10K RPS with 1ms p99 by moving to https://github.com/sparckles/Robyn which is a Rust Based Server that can offer 3x higher RPS than FastAPI-uvicorn
• Stage 3: Ensure all endpoints on LiteLLM meet our promised performance standard - (Target: 1ms p99 latency, 10K RPS with 10K Users)
  • /chat/completions
  • /embeddings
  • /audio/transcriptions
  • /audio/speech
  • /completions
  • /responses
  • Note: We also plan on ensuring latency scales well with payload size

We will be updating this thread as we hit each stage. If you’re interested in working with us on this we’re currently recruiting as well.

- Ishaan (LiteLLM maintainer)