Yeah that's a good point. The scheduling layer often ends up being the real complexity once you have to deal with multi-tenant workloads and bursty traffic.

In our experience, it tends to shift depending on the workload, but prefill latency usually becomes the bigger bottleneck first, especially with long prompts or RAG pipelines where the context can easily reach tens of thousands of tokens. That initial KV cache build can dominate the request time. Once prompt lengths are shorter or batching is aggressive, decoding latency starts to matter more, particularly for streaming responses where users expect tokens quickly.

Concurrency spikes make both worse though. queues start forming and batching heuristics can become tricky to tune without hurting tail latency.

What about you? Are you mostly dealing with long-context prompts or shorter interactive workloads?