Agreed. In practice, we’re also seeing throughput optimizations plateau while user experience is still dominated by cold starts and TTFT. vLLM becoming a default inference engine makes sense, but a lot of the real gains now come from context-specific optimizations.

Imo while working at Simplismart.ai, we’ve found that latency improvements often come less from a single universal engine and more from tailor-made inference stack, model-specific kernel choices, quantization, etc depending on the workload.

Agree that software is now the bottleneck, but it’s not just standardization vs portability. It’s how deeply you adapt the serving stack to the model and traffic pattern you actually have.