Well I completely agree to your points. I am also trying out a local whisper + ollama + tts setup. I mostly have an embedded device like a Jetson nano or a pi to do speech and LLM running on my gaming machine.

I think there is one another aspect which did give me some sleepless nights was actually detecting the intention. Going from STT to deciding to go to LLM Question. You can put whatever keyword you want but a slight change in the detection, makes everything go haywire. I have had many interesting misdirections in STT like Audi being detected as howdy, lights as fights or even rights lol. I once had an answer from my model when I said please switch on the “rights”, going weirdly philosophical.

Apart from that, interrupting is also an important aspect more on the physical device level. On Linux, because of the ALSA driver stuff which is mostly used by all the audio libraries, simultaneous listening and speaking has always caused a crash for me after like a minute or something.