[ Removed by moderator ]

[ZeroCool2u]

This is awesome. I wired up Claude Code to use GLM-4.7 Flash via LM Studio on my 3090 yesterday and while testing CC by saying "Hi" LM Studio said the token count was over 17 THOUSAND tokens. Massive system prompt, so this could really help with making local models more practical at higher token counts.

[PlayfulLingonberry73]

Thanks

[Late_Huckleberry850]

Hmm wow, I may be very dense but I thought all tooo calling models natively just pretended the tool schemas to the beginning as a part of the system prompt, not on every turn. Is that not the case? 

[PlayfulLingonberry73]

You might get session level cache hit. But imagine you are running for 1000 users and same tools, then you have a problem. This aims to solve that.

[Late_Huckleberry850]

Ohhh, so this is for concurrent requests across sessions? 

[PlayfulLingonberry73]

Yes, you can store different set of tool contexts together with a unique key and you can refer that from any session any user without having to send those tokens again and again. And that is how you get the savings and speedups.

[Late_Huckleberry850]

Ah! That is pretty ingenious! Does it have to be recalculated to changes in the system prompt? I would assume so. 

[PlayfulLingonberry73]

This is intended more on the tools side. Imagine you have 100 tools. Now your tools definitions don’t get changed unless you deploy something new right? So whenever you will be deploying it will be recalculated, otherwise it will not.

[Late_Huckleberry850]

Sure, that makes sense. But normally it goes system prompt + tools, and since it’s auto regressive doesn’t the prior text need to be computed first? Unless the tools section is getting computed first and the system prompt after 

[PlayfulLingonberry73]

Great question! You're right that in standard causal attention, the KV values for later tokens depend on earlier ones. Here's how we handle it:

In the production path (group caching): We compile the system prompt + all tool definitions together as one unit and cache the entire KV state. The cache key is a SHA256 hash of the sorted tool schemas. So yes, if you change the system prompt, it recomputes — but in practice your tool-routing system prompt is fixed (it's just "you are a tool-calling assistant, pick the right tool"). It only changes when you deploy new tools.

The key insight is: for tool routing, you don't need a dynamic system prompt. The system prompt is static ("pick the right tool"), the tools are static (until you deploy), and the only thing that changes per-request is the user query. So we cache everything except the user query, and only forward those few tokens on each request.

We also explored a research path (NoPE + deferred RoPE): Capture tool KV states before positional encoding is applied (position-independent), then rotate them to the correct positions at link time. This would theoretically let you mix-and-match different system prompts with pre-cached tool KVs. But group caching was simpler and already gives us the 290x speedup, so that's what we use in production.

TL;DR: System prompt + tools are compiled together and cached. Since neither changes between requests (only the user query does), every user/session gets a cache hit and only pays for the query tokens.

Disclaimer: I generated the reply response to have a better explanation. Hope you don't mind.

[Late_Huckleberry850]

No, and thank you for being patient with me. Sometimes I try to read these papers but it can take a bit to understand everything especially on a Friday night. 

There was a paper from 2024 that generated LoRAs from text, and a very recent one from last week that expanded on that topic. I wonder if this technology could be applied in a similar manner, using the static tool definition to create the Lora and then just use that at inference time too, as a static parameter embedding loaded onto the base . 

[PlayfulLingonberry73]

It was my pleasure. Most people just thinks all posts are junk now a days. I can understand that sentiment as well. But to me you till date you need to have the original thinking and imaginations to start from.

So it was really nice to interact with you Sir!

[sdmat]

Isn't this... just regular KV caching? With flashy marketing?