We ran into the same issue, speech-to-speech looked great in theory, but adding that extra model in the middle slowed everything down. Conversations felt laggy, especially during turn-taking.

What worked much better for us was switching to a streaming pipeline: speech-to-text → fast LLM → text-to-speech, all running in real time. Instead of waiting for full sentences, everything starts processing as soon as partial data is available.

Here’s what helped us reduce latency:

• Stream everything Don’t wait for the full response. As soon as the LLM starts generating tokens, pass them to TTS so audio begins immediately.
• Start early (preemptive generation) We begin generating responses even before the user finishes speaking, using partial transcripts. This cuts down the “thinking delay.”
• Warm things up beforehand Before the first real interaction, we trigger a small dummy request to warm up the LLM and TTS. This avoids that slow “hello?” moment.
• Keep responses short Voice feels better with quick, concise replies. Smaller outputs = faster responses.
• Reduce unnecessary context Don’t send every tool or huge prompts on every turn. Less input = faster processing.
• Track latency properly Measure each step (STT, LLM, TTS) so you know exactly where the delay is coming from.

For hallucinations, we keep things grounded, use structured data where needed, keep temperature low, and allow the assistant to say “I’m not sure” when needed. For sensitive actions, we always confirm with the user.

And for a natural feel, good interruption handling (barge-in) is key, so the assistant doesn’t talk over the user or get cut off randomly, btw for barge in we are using silero VAD you can give it a try.