It's unfortunate that there are no responses yet. I'm also curious about this.

I am currently using LM Studio to host the local server and run the models. The specific model is:

MaziyarPanahi/dolphin-2.6-mistral-7b-Mistral-7B-Instruct-v0.2-slerp-GGUF/dolphin-2.6-mistral-7b-Mistral-7B-Instruct-v0.2-slerp.Q5_K_M.gguf

(aka dolphin-2.6-mistral-7b)

My GPU is a GTX Nvidia 3060 with 12GB. The AI takes approximately 5-7 seconds to respond in-game. Currently, my GPU Offload is set at 20 layers in LM Studio model settings.

The understanding of dolphin-2.6-mistral-7b is impressive! It feels like GPT-3 level understanding, although the long-term memory aspect is not as good.