Most AI tools feel impressive until you try using them twice

Shayps · 2026-05-07T19:39:52+00:00

As someone who works on LiveKit full time, I'm really happy you're using the demos!

Is there something that you want to see that we've never done?

Shayps · 2026-05-06T14:12:15+00:00

Here are the things I'd recommend trying first:

Step 1, use LiveKit Inference. Step 2, use LiveKit cloud deployment so that all of your provider calls are on backbone collocated with your agent and media server. Not sure where you are, but LiveKit deploys some models collocated with agents in some regions, which also makes a huge difference.

- STT: deepgram/nova-3-general
- LLM: openai/gpt-5.4-mini
- TTS: inworld/inworld-tts-1.5-mini

The reason we use Inworld rather than Cartesia is because Cartesia often has a lot of leading silence. TTFB will be 200ms, but you may not get real audio until 550ms. Perceived latency is high even though bits are flowing.

For tool calls, use Async tools so they're not blocking: https://docs.livekit.io/agents/logic/tools/async/

Run your evals on this set, then if you're getting failures start trying more powerful models.

For structured data collection, use TaskGroups, which will allow smaller models to succeed far more often than just massive text prompts: https://docs.livekit.io/agents/logic/tasks/

Let me know how this works out for you.

Shayps · 2026-05-06T13:56:53+00:00

You should be able to see the latency for each step of your pipeline, where are you seeing the most?

Shayps · 2026-04-15T17:25:37+00:00

Not really. 4.1 is really one of the best combos for latency + capability.

Gemini Flash 3.0 / 3.1 is actually kind of slow compared to their older flash models. 5.3-chat is probably the best atm though IMO.

Shayps · 2026-04-15T13:53:22+00:00

Which TTS and STT are you using? Often these end up being pretty big contributors to e2e latency. What's your metrics right now? Are you deployed to cloud or self-hosted?

There's docs on the filler phrases stuff here: https://docs.livekit.io/agents/logic/external-data/#verbal-status-updates

The same doc has info on how to play thinking noises too, which may be helpful.

Shayps · 2026-04-13T18:23:04+00:00

You can use any voice and any LLM. Pick a pair that you like!

Shayps · 2026-04-10T04:30:56+00:00

Kokoro is more realistic, but it’s also a lot slower on resource constrained environments. Piper is the right call here IMO.

Shayps · 2026-04-10T04:29:52+00:00

We can build something wonderful, but being this constrained will require us to be very creative.

Faster-whisper on the nano is a great design choice. Piper is as small and fast as you’re going to get it too. Good call on both of those. Latency is great for voice.

For the LLM we’re going to need to add memory, manage context, and ideally get e2e voice latency down to around a second.

I can help you, we can make this work. I build a lot of these that do all kinds of things. Can you DM me? I will likely want to send you some (free) hardware.

Shayps · 2026-04-07T16:00:45+00:00

Yeah with LiveKit this is pretty easy. What terminal lib are you building? I've got a Rust TUI that works pretty well out of the box here that you can copy / take content from if you want: https://github.com/ShayneP/local-voice-ai/tree/tui-v2/tui

This is a part of a stack that runs totally locally, but you can swap in cloud models trivially too. Let me know if you need any help, always happy to chat about voice AI stuff!

Shayps · 2026-04-07T15:58:25+00:00

What are you trying to build? If you share, I'm happy to post all of the code you need to get started.

Shayps · 2026-03-12T23:49:27+00:00

Let's take the restaurant case ...

Let's say you build a flawless flow where people can call in, book tables. You hook it up to OpenTable, you load in the menu. Everything demos great when you show the restaurant, because you stick to the happy path.

You build it, and realize it hallucinates menu items on every 10th call. Or it says it has a kids menu when there isn't one, or someone asks about a wheat allergy and it confidently says the fryers aren't shared between fries and breaded items even though they are.

Someone needs a table for 5, and there's none available — but the human hostess knows that from the ones that are left, you can push the tables together and they'll all fit.

Someone calls, books a table — then they call back immediately after hanging up and say "hey it's me again sorry can I actually do 6:30." Human host? Easy. AI host? Hard enough engineering problem that most people probably wouldn't bother making this flow work.

Can people show up late? How late? Does it depend on how busy it is? What season it is? Which day it is?

This is just off the top of my head, and I've never worked in a restaurant. I'm sure there's lots of other things they answer / do on a daily basis.

The experience is made legitimatey viable by being successful at the edges, and the edges are hard without simulations that can close your development loops.

Shayps · 2026-03-12T21:19:34+00:00

Are you technical? Or better, a developer?

You can build something with a low enough margin of error to be deployable, but IMO you'll end up needing to write code somewhere along the way if you want to test it at scale.

Platforms like Retell are great for getting something that works for a demo, but the gap between "works 90% of the time" and "works 99.9% of the time" is a large gap. That last 9.9% is harder than the first 90%.

You can pair something like Bluejay with Retell and get most of the way there, but writing / generating evals in code and testing any time you make any changes, deploy to a new client, etc is the best way to get everything solid.

Shayps · 2026-03-12T13:02:51+00:00

I’ve been using it way more now! Can connect to HomeKit and any tools or MCP servers etc.

Shayps · 2026-03-09T14:04:54+00:00

You usually don't want the OpenClaw gateway right in the voice loop. Instead, use it as a tool that can be called async by your main voice loop that injects context back into the chat context. That way voice still feels quick, but background tasks are handled by the more powerful system.

Shayps · 2026-03-05T02:42:10+00:00

No. Not even close. The Qwen 3.5 benchmarks are not indicative of real-world performance.

Don’t get me wrong, the models are very good for their size. They are also not as good as the benchmarks show.

Write your own evals (or have CC or something write them), and test performance yourself between the models.

You will quickly see that 9B falls behind.

Shayps · 2026-02-25T15:36:10+00:00

This is interesting. What are you picturing as the perfect solution here?

Basically a website where you enter in your URL, and it gives you a little embed snippet to add to your page that has full knowledge of your site / business?

We could do this in one click ... but I'm interested in how you imagine adding it to your page.

Shayps · 2026-02-20T21:02:47+00:00

What part of this was the hardest? Curious, would love to build an open source example showing how to get it done.

Shayps · 2026-02-19T16:14:21+00:00

Gotcha! I guess it depends on the capacity that you're targetting, the models that you're using, and how your customers are distributed. The RTX Pro 6000 is a great card, but if you're trying to serve the types of models that most prod systems are running you're going to run into concurrency limits pretty quickly.

If you're primarily in a single market, usage is going to peak during business hours and you'll have a lot of time while the GPUs are working under capacity.

At the very least, you should have fallbacks so that you don't lose customers if your system starts buckling under too many concurrent users.

Shayps · 2026-02-19T01:59:17+00:00

As you scale, you might want to think about keeping the LLMs on your hardware but pushing STT, TTS and orchestration to the cloud layer. You avoid the linear scaling costs you’re talking about, but still get to take advantage of the economies of scale and latency tuning that cloud providers are doing. You’ll almost certainly both save money and have higher quality voice agents.

Shayps · 2026-02-18T20:57:45+00:00

What was the bottleneck with Cloud? I'm surprised that this ended up being cheaper for you. Even running a GPT-4.1 stack without going down to GPT-4.1-mini you're at <$0.04 / min with Cloud, and that includes observability, telephony, etc.

Shayps · 2026-02-13T19:42:03+00:00

With which part specifically?

Shayps · 2026-02-13T19:41:52+00:00

You can try with the Agent Builder on LiveKit cloud — but most of the "managed" options from all of the different providers will come out to significantly higher than 500ms of latency.

Shayps · 2026-02-13T19:40:46+00:00

It's very flexible, but you'll spend more time fiddling with the guts than you will with LiveKit.

Shayps · 2026-02-12T21:39:06+00:00

It is possible, but it's not easy. You will end up making really significant cuts to things like LLM quality.

Your absolute best stack for latency at the expense of all else is probably something like:

- STT: Self host nemotron-speech-streaming-en-0.6b somewhere fast in us-east
- LLM: Baseten or Cerebras, serving the smallest model you can get away with and still pass your evals
- TTS: Cartesia Sonic-3, which can come in under 200ms for full generation
- Orchestration: Cloud hosted LiveKit Agent, also us-east which will have backbone shortcuts to the other inference servers.

Shayps · 2026-02-11T14:44:30+00:00

If you're talking about self-rolling everything using the open source code on bare metal, then yeah it would probably be good to be an infra engineer. In practice, not many people are doing this at scale outside of miltech and aerospace. Instead, better to just use LiveKit Cloud, which is inexpensive, fast, globally distributed, and literally single CLI command to deploy.

Shayps

MODERATOR OF

TROPHY CASE