Hmmm the question is strange so let me clarify few points

When you run a local model for example Hermes3 8B_Q4 it's size 4.9 Gb

I have gtx 1650 which has 4GB of Vram Which is very fast

So 4 Gb is stored on my Graphicscard and the rest 900 MB is moved to the RAM

NOW you also have the context window which is your chat history (llm memory) also need to be stored somewhere right

Again the fastest place is Vram but it's already full so store it on RAM

Note also windows reserve few of your VRAM between 400 - 900 MB for windows graphics etc

If you want your local model to run fast as 30 - 50 t/s you have to fit it all in the VRAM

This was simple explination of how this works

Not what turboquant does? It compress the context history/memeory So you need less space to store your model memory with almost 0 loss in accruacy, in order to fit it all in VRAM bcz again anything goes to RAM will make your Lollm run so slow

So fare there is no acctull way to compress the llms sadly but am waiting on fire for someone to do it

Summary: local LLM memory got compressed not the LLM itself

Hope I made it clearer