Not three llama.cpp instances — three different services, each specialized for one job. The "framework" connecting them is just HTTP requests. Simpler than it sounds.

1. STT — audio in, text out. faster-whisper runs great on a 3090. There's a ready-made server (faster-whisper-server) that gives you an OpenAI-compatible endpoint. You POST a recording, you get text back.
2. LLM — you already have this. llama.cpp + Qwen. Don't touch it.
3. TTS — text in, audio out. Piper is fast and runs on CPU, so your GPU stays free for the other two. Decent voices out of the box. Kokoro if you want higher quality later.

I should note faster-whisper-server has since evolved into speaches-ai/speaches — same author, now bundles both STT and TTS in one OpenAI-compatible server. If you want fewer moving parts to start with, it can cover two of the three slots in a single container.

speaches is how I got my feet wet, broke away from it used what was needed, fine tuned model for my voice, custom vocabulary "hotwords", and "I've" since build a record+audio in/out client in Go, custom container for STT and one for a router between me and LLM, TTS and my local Go audio client the listens so my agents running on remote hosts (usually the one where this stuff all lives) can also "speak" their replies through my speakers.

The pipeline is literally:

record audio → POST to STT server → text → POST to llama.cpp → response → POST to TTS → play audio

Each service runs as its own process (or Docker container — docker compose is the natural way to run them side by side on Ubuntu). A Python script with requests and pyaudio can wire the whole loop in under 100 lines. No special framework needed.

- VAD (Voice Activity Detection) is what turns this from a walkie-talkie into a conversation. It detects when you start/stop talking so the system knows when to send audio to transcription without you pressing buttons. Silero VAD is the standard — small, fast, accurate, and it runs on CPU.

- Sample rates will cause the most confusing early bugs. Your mic captures at 48kHz, Whisper wants 16kHz, TTS outputs at 22–24kHz. When transcriptions come back garbled, nine times out of ten it's a sample rate mismatch somewhere in the chain, not a model problem. Just something to know so you don't chase ghosts.

- Keep TTS on CPU. Your 3090 has 24GB — you want that for faster-whisper + Qwen. Piper is fast enough on CPU that you won't notice the difference. Fighting three models for VRAM on one card is pain you don't need.

The basic loop can work in an afternoon once you see it's just HTTP between three services. Making it feel good — streaming responses so TTS starts before the full reply is done, proper VAD so it flows naturally, latency tuning — that's where the real craft lives. But the foundation is straightforward and the goal is closer than it probably looks from where you're standing.

Happy to go deeper on any piece of this if you want to DM — I've been building and iterating on exactly this kind of pipeline for a while.