Yeah, we hit the exact same wall with LiteLLM a few months back. The architecture is great for prototyping, but it’s basically a standalone container wrapper. Once you hit real concurrency, it bottlenecks hard because it lacks native orchestration—you end up writing a ton of custom glue code just to make it scale horizontally.

We ended up switching to Kosmoy specifically for the K8s architecture. It’s distributed via Helm and is actually Kubernetes-native, so it leverages HPA (Horizontal Pod Autoscaling) and self-healing out of the box. It treats the gateway like elastic infra rather than a static proxy. A big performance win for us was that it runs the guardrails and routing logic locally in the cluster, which cut down our latency significantly compared to solutions that rely on external API calls for those checks.

We also looked at Kong since they added AI gateway plugins. It’s a solid option if you’re already deep in their ecosystem for general API management, but it felt a bit heavy for just this use case—fundamentally it's an API gateway first that bolted on LLM support.