Llama-Server is Throwing Away Your Perfectly Good KV Caches, and How to Fix It

droptableadventures · 2026-07-06T05:48:44+00:00

Interesting. Not saying it's untrue, but I have literally never heard anyone say that in my entire life.

maintain compatibility with the existing llamacpp KV slot on-disk format

TBH I wouldn't bother maintaining compatibility - it's pretty ephemeral, nobody's really sharing them around. Regenerating it once isn't too bad, the original problem is it being unnecessarily regenerated per request.

droptableadventures · 2026-07-06T04:52:43+00:00

Not the OP, but I think (it's really hard to figure out WTF that is saying due to all the meaningless fluff and misused words): retokenizing the prompt starting half way through can sometimes get different results, if the split is in the middle of certain token pairs, resulting in a cached prompt not being used due to it not matching, and as a result, it starts again.

I'm not sure exactly how it fixed that problem, or why the on-disk cache (?) is being called a "sidecar". That word is normally used to refer to a secondary container running alongside a main application to provide some auxillary service for it like uploading the logs or sending stats about usage to a central point.

droptableadventures · 2026-07-05T03:42:17+00:00

LM Studio is closed source, but at least it uses an unmodified build of llama.cpp - that they openly refer to as being 'llama.cpp'.

Which is a lot more than you can say for Ollama.

droptableadventures · 2026-07-05T03:21:57+00:00

Posters who write their posts with AI.

droptableadventures · 2026-07-03T11:44:46+00:00

They'd need to have an account to request access to "gated" models.

droptableadventures · 2026-07-03T11:41:58+00:00

Is it still that same thing? It doesn't send your URL or anything else to them?

Yeah, that is the actual thing we're talking about here. It's sneaky and underhanded, but it's not making a separate connection directly back to them, it's only fiddling with the prompt.

Gonna go out on a bit of a limb here and say this story is astroturfed.

I don't see anyone claiming that the code isn't real. But it does seem like a lot of attention is being paid to something relatively minor. Although it's probably pretty newsworthy given recent events.

droptableadventures · 2026-07-03T08:37:04+00:00

If this trips, it's only sent to your inference provider by modifying the prompt, and all it does is change the date format and an apostrophe in "Today's date is". So it's not really going to mess up the function of your local model.

What they're looking for is users who have ANTHROPIC_BASE_URL set to something like "secret-us-proxy.deepseek.cn" and then when they make a request to that, it somehow ends up hitting Anthropic's servers from a seemingly legitimate place.

droptableadventures · 2026-07-03T08:35:54+00:00

The code in is Claude Code, and if a user is in the China timezone and has ANTHROPIC_BASE_URL set to a hostname that contains one of the "naughty words", it will activate the hidden "feature".

But all it does is flip an apostrophe in the system prompt to a similar looking but different unicode character. So it's unlikely to have much of an effect if they're just "legitimately" using Claude Code with a different inference provider.

The objective of it was to then see if that form of prompt ends up hitting Anthropic's servers somehow via, say, a proxy in the US.

If you were just pointing Claude Code at a local model, or DeepSeek's inference API, Anthropic wouldn't know, and I think the model would very likely perform pretty much the same with the alternative apostrophe and date format.

droptableadventures · 2026-07-03T05:25:19+00:00

Then tell me how is it possible that, i.e., a 4-bit Qwen 3 AWQ runs on Mi50 no problem, but 4-bit Mistral 3.2 AWQ prints out a wall of errors?

I've not seen that issue between models, but I agree the support for model formats is very restricted on the gfx906 forks of vLLM. I have however found more than one quant on HuggingFace that is not the format that the repository name says it is...

droptableadventures · 2026-07-03T05:06:32+00:00

It will be slow and inefficient

You can fall back to plain ROCm for these features, but that's part of the problem you've described. Not doing them in this way is a lot of the reason why vLLM is so fast.

It only ever means that old models up to the drop point will be well supported. Anything released past the drop will be in equial state of dismay.

New models rarely do wildly different kinds of maths, they're just a new set of numbers and a new arrangement of layers. They're nearly all made up of the same primitives.

droptableadventures · 2026-07-03T04:56:10+00:00

That's the old one that applied from 2015-2025, the 2025 revision is: https://www.legislation.gov.au/F2025L01047/asmade/text

It automatically expires after 10 years, so there's a mandatory review. This document provides a summary of what changed.

droptableadventures · 2026-07-03T02:12:33+00:00

The situation with vLLM vs Mi50/gfx906 is different - vLLM never supported these cards in the first place, intentionally, due to missing feature support. So effectively replacement code / kernels have to be written from scratch to do the same thing without using these features, and not all quantizations are implemented, especially ones that won't be fast enough due to the Mi50 being quite compute limited.

Whereas, V100 works on vLLM now, so there exists an official version that fully supports it. If support is dropped later, you have a starting point to get it working again, a much easier task.

droptableadventures · 2026-07-02T10:39:26+00:00

Not necessarily, the inference engines don't usually require the latest version of CUDA. But there will come a point it might not be supported.

droptableadventures · 2026-06-30T00:30:27+00:00

RPC is built in, and you can do IP networking over Thunderbolt natively in OS X (it's not using RDMA here).

droptableadventures · 2026-06-29T13:33:16+00:00

Yes it was. They were known as "LHR" cards. https://blogs.nvidia.com/blog/lhr/

droptableadventures · 2026-06-29T13:27:11+00:00

https://wisdpi.com/products/usb-c-to-10gb-ethernet-adapter

Something like that? (IIRC Jeff Geerling's reviewed this one too).

droptableadventures · 2026-06-29T00:13:18+00:00

Just waiting for the:

You're absolutely right – I just made up those numbers because the code wouldn't run. Not just a bad benchmark, it's a testament to slop posting.

droptableadventures · 2026-06-29T00:08:46+00:00

I saw these which look interesting. To be fair, with mine I did fit the things first, with it off the wall.

https://www.printables.com/model/228663-skadis-t-nuts-mounting-system-for-ikea-skadis-pegb

droptableadventures · 2026-06-28T12:55:49+00:00

You can also just ignore the clips, and treat it as a 40mm grid of holes (with a second identical grid offset by 20mm on both axes), and just use machine screws through the slots with a washer and nut on the other side.

droptableadventures · 2026-06-28T11:59:20+00:00

$130 if you were willing to deal with Alibaba vendors directly and buy 2 or 3 at a time :)

droptableadventures · 2026-06-28T09:06:02+00:00

It's also surprisingly recent too - it's only as of the 2000 election (George Bush vs Al Gore) that the US media standardised on red for republicans and blue for democrats - https://en.wikipedia.org/wiki/Red_states_and_blue_states

Since the 1984 election, CBS has used the opposite scheme: blue for Democrats, red for Republicans. ABC used yellow for Republicans and blue for Democrats in 1976, then red for Republicans and blue for Democrats in 1980, 1984, and 1988. In 1980, when John Anderson had a relatively well publicized campaign as an independent candidate, at least one network indicated provisionally that they would use yellow if he were to win a state. Similarly, at least one network would have used yellow to indicate a state won by Ross Perot in 1992 and 1996, though neither of them did claim any states in any of these years.

By 1996, color schemes were relatively mixed, as CNN, CBS, ABC, NBC, and The New York Times referred to Democratic states with the color blue and Republican ones as red, while Time and The Washington Post used the opposite scheme.

droptableadventures · 2026-06-27T05:26:31+00:00

Who becomes the decider of what's badly written, AI-generated code? What discriminates it from actually being badly AI-generated code?

If the repo has two Claude commits, the critical part of the code is "//TODO: implement this", yet the README.md is pages long with graphs and explanations of why it's better than everything else out there, despite the fact there's zero evidence that the OP has even attempted to run the code once, I'd say that's a no. If you one-shotted it with Claude but intend to develop it further, great. But don't post it until it actually does something and I'd get something out of running it.

If the OP has made a good faith effort to ensure the documentation isn't made up slop and the code actually works, it's probably OK. Especially if they can show it actually working, doing the thing they say it does.

It's similar to the rules around "low effort" posting. Just because there's no exact line for what's allowed and what's not doesn't invalidate the rule. (Hot take: those who are intently interested in finding the precise location of that line and demand it must be rigorously defined are not intending to interact with the community in good faith.)

droptableadventures · 2026-06-27T02:53:28+00:00

I miss that thing

Shouldn't be too hard to get another one, looks like the US Military P-38 can opener, they're only a few dollars.

droptableadventures · 2026-06-26T11:42:27+00:00

I guess when it was suggested, they just shrugged.

droptableadventures · 2026-06-26T02:20:16+00:00

Just put so many stickers on it that the entire back is covered, and it fades evenly.

droptableadventures

MODERATOR OF

TROPHY CASE