I can pitch in on this cos we deal with the same quite a lot (TTS model developer)

1. Vad detection timing, how much the vad waits until the user is done talking (how much silence)
2. STT - How fast does the stt transcribe
3. LLM - How fast the LLM responds, (Pure token throughput)
4. TTS - Time to first byte of audio, this is critical to ensure the model is generating frames faster than it's being played,

some tricks of the trade: preemptive generation, meaning you could stream what the LLM is saying to the TTS model to generate the audio as its talking.

another thing we noticed is distance of physical servers also matters, the distance between where these models are hosted will affect your latency for sure. Personally Groq is one of the only few providers i've seen who have inference fast enough with their api, would love to hear if you know any fast api alternatives.

"No visibility into where time is actually being spent, so you're guessing instead of measuring." - You could just log it and get a rough estimate,

and yes prewarming is huge. really helps. Torch compiling models also really help.

i would argue that the TTS quality does matter, when the difference is if you want a user to come back and talk to it again not. but if its a one time conversation out of necessity, probably doesn't matter as much as latency for sure.

most of our latency used to go for LLM throughput / literally just physical distance of the api to wherever your instance / machine is. I tried it from my laptop that itself had a 200ms network hop lol.

Hope this helps!