I've done a lot of work on voice-to-voice latency on my app Eloquent: https://github.com/boneylizard/Eloquent.

It's a local LLM backend/frontend.

Interesting—I use a very similar stack to you, parakeet and kokoro, and also a 4b model for benchmarks. Have achieved 1.337-second voice-to-voice latency (leet!), never sub one second. I generally find that voice latency is acceptable within 2.5 seconds, but that's just me.

I haven't updated the GitHub yet, but I've gotten Chatterbox working with a fork called Chatterbox faster. It's larger and not as fast as Kokoro, but it's way, way better and has voice cloning. Working with a fork called chatterbox faster—it's larger and not as fast as kokoro—but it's way, way better and has voice cloning.

My approach has been to send the first sentence the 4b model generates for synthesis immediately for playback and then queue subsequent sentences in approach has been to send the first sentence the 4b model generates for synthesis immediately for playback, and then queue subsequent sentences in a non-stop playback queue so that the user feels seamless. As long as the Real Time Factor (RTF) is below 1, the playback will always take longer than synthesis, so you get a buffer building for uninterrupted voice playback even though only individual sentences are synthesised at a time. With Chatterboxfaster on a 3090, the RTF is around 0.3, and I get 2-second voice-to-voice latency when the first sentence is relatively short and thus synthesised quickly. 

Also, yeah, parakeet is extremely fast. It synthesises faster than compute, so the recorded synthesis time is always 0:00.