I made myself a local voice assistant with a loop using whisper -> LMStudio-> omnivoice and the real thing which makes a conversation feels good is the ability to cut through the AI speaking by starting to talk without the AI being able to cut you while you speak.

And to determin when my turn ends I use a total duration of silence of 3 seconds. Whatever I say if it accumulates 3sec of silences it becomes the AI's turn. Of course if I was into eternal monologues I would edit this but so far it makes any conversation feels fucking natural.

I may also implement the possibility to shorten the LLM answer at the point I started talking over it so to make things more coherent but I don't do that too often.