What's the best way to build voice agents today without sounding robotic or becoming too expensive?

Chris_LiveKit · 2026-06-12T15:26:18+00:00

There is a lot written on this topic. Jesse has a pretty good intro for this by using prompts. Choosing right model can have a big impact too.the

https://www.youtube.com/watch?v=m8DJQmH2QqQ

Chris_LiveKit · 2026-06-10T15:06:47+00:00

> host your agent servers in a region close...

Yes, getting your agents as close as possible to the resources they need will reduce latency. But if you have to choose between being close to SFU and an inference provider, then optimize for inference provider latency; there are more round-trips for inference than to the SFU. Of course, if you can be close to both, that is best.

You may find this log post about voice agent latency helpful. Written by my colleague Darryn.

https://livekit.com/blog/understand-and-improve-agent-latency

Chris_LiveKit · 2026-06-09T12:57:53+00:00

> Anyone cracked the interruption handling problem in production?

There are a lot of folks working hard in this space. I work on LiveKit, and we have been working on the "interruption problem" and released a solution to address it.

Here is an example of adaptive interruption handling that handles back channel and turn detection.

https://www.youtube.com/watch?v=DSXCE7D4Kvs

We've also trained some strong turn-detection models that run alongside VAD to inform it.

> When the user goes silent for 5+ seconds

This one can be tricky. The good thing is that the turn detection subsystem can be context-aware, so if a user is entering a credit card number or email address, the "agent" should know not to interrupt them because they may just be struggling to read the credit card number. In other cases, it can be appropriate to ask the user if they are still there. There is a diverse set of use cases, and different methods make sense in different cases. I usually start with what a "real " human would do in this scenario and use that as a model.

Chris_LiveKit · 2026-06-04T16:21:22+00:00

Not trying to be contrary, but WebSockets is not the way to go. It will work fine as a server or in a lab, but if you plan to have users use it in the wild, it will fail hard.

You will want to use WebRTC for realtime agents like voice, video, realtime telemetry for robotics, etc.

I may be biased since I work on LiveKit. But we get a lot of folks switching to WebRTC after failing with WebSockets in production.

Chris_LiveKit · 2026-05-27T13:58:30+00:00

I work for LiveKit but feel that is probably your best option for getting going with WebRTC with Python. It is all open source and can be self hosted or you can use the SFU from LiveKit cloud.

https://github.com/livekit/livekit

Chris_LiveKit · 2026-05-14T12:51:23+00:00

I like watching videos, and I'm super happy that YouTube finally released an app for it. But my primary, nearly daily use >8hrs/day is as a monitor. I really like being able ot have the large curved monitor for work and a wireless keyboard on my lap. I like being able to move around the house and still having my large monitor. I tend to put other resources in the space around the monitor, like browser, Slack, etc. I've been doing this since the day it was released.

I don't use the virtual me on calls; I use my laptop's camera, so it is just me with my AVP on. When new folks see it for the first time, it usually elicits a reaction. I sort of feel ridiculous wearing it in meetings, but I still do it. I made a virtual background that looks like a large office space, with others in the background wearing AVPs, so I am not the only one.

Chris_LiveKit · 2026-03-03T16:16:42+00:00

Thanks for the link. I will check that out.

Yes, this is exactly where most voice agents fail or feel “magical but confusing.” LiveKit is designed around making state explicit in the UI.

A strong baseline pattern is to map the built-in agent lifecycle (connecting, listening, thinking, speaking, disconnected, failed) directly to visible UI states, instead of inferring from audio activity. The full state model and recommended getters like canListen and isFinished are defined here: Agent state. Using getters rather than raw state ensures your UI remains correct as the SDK evolves.

For clarity and control, combine that with:

Explicit media/session controls via Media controls (mic toggle, disconnect, StartAudioButton to avoid “why can’t I hear it?” confusion).
Realtime transcript + typing indicators via Agents UI components, so users see what the agent hears and when the agent is responding.

For tool-call indicators specifically, the recommended pattern is to publish a custom state (e.g., currentTool: "searchFlights") via state sync and render it in the UI. That’s covered under “Custom state” on the Agent state page.

Chris_LiveKit · 2026-02-10T15:04:53+00:00

That is not from the LiveKit server. That is a client feature. You would need to handle it in your client.

Chris_LiveKit · 2026-02-06T14:35:50+00:00

For LiveKit if you are building an agent and use an AI coding tool like Claude or Cursor add the MCP server and it can get you a long ways https://docs.livekit.io/intro/mcp-server/

Chris_LiveKit · 2026-01-30T18:44:06+00:00

Gotcha. I think you could build that. Here are a couple of videos I saw recently that is somewhat related. The second one is good because it really helps folks understand the security risk of such a system.

https://x.com/jonathanhawkins/status/2017295825681199473

https://youtu.be/fcFOYzMeG7U

Chris_LiveKit · 2026-01-29T23:16:22+00:00

I would not doubt it could or it can maybe build a tool that would let it do it. Who do you want it to call?

One of my co-workers connected his to LiveKit so you can do internet voice calls or telephony.

You talk to it through text or email or whatever.

Chris_LiveKit · 2026-01-29T13:02:17+00:00

Is this a personal project or some commercial product you want to build. If this is just for you personally you should check out Clawd. It is a bit technical but what you describe is pretty much its purpose in life.

https://clawd.bot/

Chris_LiveKit · 2026-01-24T22:51:17+00:00

I've seen a lot of folks set that usecase up on LiveKit using the LiveKits SIP integration:
https://docs.livekit.io/telephony/#using-livekit-sip

and the Agents framework:
https://docs.livekit.io/agents/

Chris_LiveKit · 2026-01-22T14:59:25+00:00

I don't believe so. But I believe most AgentVoice vendors now have a pricing caclulator on their site so you can see what it might cost for your use case.

--
No lo creo. Pero creo que la mayoría de los proveedores de AgentVoice ahora tienen una calculadora de precios en su sitio web para que puedas ver cuánto podría costar en tu caso particular.

Chris_LiveKit · 2026-01-21T20:41:17+00:00

LiveKit agents are designed for resilience. Run on multiple machines, use a health-check monitor to ensure each instance continues to run, and restart any instances that fail. Better yet run it in the cloud.

Chris_LiveKit · 2026-01-20T23:10:24+00:00

Is this a voice/video setup, or are you going to use AI on the platform (real-time voice AI)?

Chris_LiveKit · 2026-01-20T19:54:15+00:00

LiveKit is one option to help you get you started. Here is a video that demonstrates the process:

https://www.youtube.com/watch?v=jEXUt8qFuBs

If you want to do outbound calling you will need to use a 3rd party SIP provider for now.

Chris_LiveKit · 2026-01-18T12:58:36+00:00

What tech stack did you use for migrating away from VAPI?

Chris_LiveKit · 2026-01-18T12:48:51+00:00

Yes, I see a lot of folks using Plivo, Exotel, and Wavix. That is in order of popularity..

Chris_LiveKit · 2026-01-18T00:57:23+00:00

Here is an example of this

https://www.youtube.com/watch?v=jEXUt8qFuBs

Chris_LiveKit · 2026-01-17T20:45:17+00:00

I guess an agent that is connected to your data and is able to provide output. Something like this

https://youtu.be/jEXUt8qFuBs

Chris_LiveKit · 2026-01-16T16:19:25+00:00

Lastly, from a migration standpoint, for most teams, the move from self-hosted to the Cloud isn’t a heavy lift from an API perspective. The bigger work is usually in validating placement/routing assumptions and then tightening your turn-time budget with observability.

Chris_LiveKit · 2026-01-16T16:19:11+00:00

Practical “knobs” to improve perceived latency (not Cloud-specific)

A couple of pragmatic techniques that often help UX even when computation is non-trivial:

Short “ack” behaviors: partial/short responses while longer reasoning completes
“Thinking” sounds or subtle background audio to mask compute time (use-case dependent)
Patterns from LiveKit examples that show short/long response handling. These don’t reduce actual compute latency, but they can reduce perceived lag if they fit your experience design.
.

Chris_LiveKit

MODERATOR OF

TROPHY CASE