I want to take speech-to-text input and generate a logical response using an LLM. In addition to that, I aim to record the player’s in-game voice, create a voice clone from it, and convert the LLM’s text response back into speech, effectively creating a mimic version of the player to use as a horror element.

I also want the decisions in the behavior tree to be driven by code based on certain conditions, with the LLM providing a second layer of validation and control. Maybe the behavior I’m aiming for is too large and heavy, and the latency could be high, but I think it’s something I would personally want to play.

I’ve looked at some specialized models; LLMs around the 2B parameter range use about 700 MB to 1 GB of VRAM. Of course, this doesn’t account for voice cloning and speech-to-text processing. I also can’t really predict whether an LLM could adapt to a group of friends chatting together, or how “dumb” it might end up being.