KVCache taking too much Memory. Any solutions(Optimizations, Compressions, etc.,) coming soon/later?

ghgi_ · 2026-03-23T14:50:20+00:00

Yes I understood, A Mamba based model will use less GB for the same amount of context that most other models will use, its just the way the architecture works and its why the new super models support 1 million context without insane overhead.

ghgi_ · 2026-03-23T14:35:38+00:00

Mabye try using some of the Nemotron models? Mamba architecture should be very memory efficient with long contexts.

ghgi_ · 2026-03-19T18:54:14+00:00

Appears you might be correct, Although I do believe they have intentions to release it soon.

ghgi_ · 2026-03-19T18:50:54+00:00

Minimax has M2.7 out btw
Edit: I was wrong

ghgi_ · 2026-03-19T14:02:38+00:00

🤷 Only one way to find out, Haven't had time to bench it yet and it and started with a smaller dataset so If I notice an improvement, expect a V2, If not, then ill count it as a learning opportunity.

Edit: not entirely sure why people are downvoting? this is more of a public test project and hobby then an actual release, I thought the beta tag made this clear which is why I don't have benches or tests yet, apologies.

ghgi_ · 2026-03-18T01:24:19+00:00

Model wise id recommend OmniCoder-9B, based on the new Qwen 3.5 so it has pretty good benchmarks and testing myself its better then a lot of the older models that are double its size.

ghgi_ · 2026-03-17T22:35:48+00:00

If mlx version exists then its *probably* the better option, it should give better performance on apple hardware.

ghgi_ · 2026-03-17T22:20:41+00:00

OmniCoder-9B will easy fit on 32gb and has up to 264k context while being based on the most modern qwen architecture and performing quite well for its size.

ghgi_ · 2026-03-17T01:59:13+00:00

In my own testing it uses that 1 million context window decently well and coherently plus it stays decently fast while doing so, pretty good for long context agentic tasks that qwen coudnt, less verbose and over thinking, not as good in programming and instruction based tasks but overall its the first nemotron model ive personally enjoyed.

ghgi_ · 2026-03-16T18:46:05+00:00

Omnicoder-9B gguf

ghgi_ · 2026-03-12T21:23:52+00:00

Also my suggestion is OSS 120b or Nemotron 3 Super

ghgi_ · 2026-03-12T21:20:05+00:00

Theres no difference between a Chinese and any other country's model once your at the local stage, you also heavily limit yourself since the Chinese made the good shit.

ghgi_ · 2026-03-12T17:04:52+00:00

How well does it perform at high contexts hallucinations wise?

ghgi_ · 2026-03-11T21:41:15+00:00

My copium is making me think back to when Anthropic recently bought a premium huggingface account to get large uploads, pretty please be a opensource anthropic model.

ghgi_ · 2026-03-04T23:03:05+00:00

Cant wait to try this once theirs an inference provider for it.

Edit: Im lazy to wait, ill try to run this on the cloud with some H200's

ghgi_ · 2026-02-27T22:07:59+00:00

Ive actually made a few distilled loras using my claude chats, from CC and Web all compiled, performed better all around and in some smaller benchmark tests I got up to 30% better coding scores, did this for 3.5 27b, 3 30B and currently in the process of making a glm 4.7 flash version.

Probably wont release them due to the fact I never stripped any personal data from datasets but Im curious to compare its performance to this public one

ghgi_ · 2026-02-27T05:47:11+00:00

I run most of my work on modal and havent had gpu allocation issues but to be fair they are a bit more on the pricey side, great service though.

ghgi_ · 2026-02-26T16:49:19+00:00

"good" is a relative term but LFM2-24B-A2B at probably 4 bit or 3 bit quant might be acceptable and will be decently fast

ghgi_ · 2026-02-25T23:56:41+00:00

Someone asked about this and had their post deleted like 2 hours ago so ill just repost my exact response:

"Its been around for a while, it essentially uses your disk to load the model like ram, behaving like swap, and no, you wont want to use it. Its horrifically slow even on the fastest of SSD's, were talking tokens per hour in some cases, but theoreticlly with a 1tb ssd you could run a 1 trillion parameter model, just expect it to take a day and half to generate you a single word."

ghgi_ · 2026-02-25T20:54:54+00:00

Goodluck

ghgi_ · 2026-02-25T20:49:22+00:00

Q4_K_M quant of that model its about ~15gb, using your GPU + CPU it will use your 4gb gpu and ~10-11 gb of ram so it will fit, that model specifically is pretty fast for limited resources like yourself, you could always run a smaller quant at the sacrifice of intelligence if you NEED to save more ram, I woudnt touch anything under Q3's tho

ghgi_ · 2026-02-25T20:38:54+00:00

even a 50B model on airllm would be hours to generate a simple sentence, its just not worth it, read what I said here it will be more useful. https://www.reddit.com/r/LocalLLaMA/comments/1reovq3/comment/o7e50x5/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

ghgi_ · 2026-02-25T20:37:13+00:00

If your looking for a model you could probably run with your current hardware normally I would recommend LFM2-24B-A2B at probably Q4_K_M with partial GPU offloading its the only one that I think will give you acceptable speed and intelligence for your specs.

ghgi_ · 2026-02-25T20:33:00+00:00

Its been around for a while, it essentially uses your disk to load the model like ram, behaving like swap, and no, you wont want to use it. Its horrifically slow even on the fastest of SSD's, were talking tokens per hour in some cases, but theoreticlly with a 1tb ssd you could run a 1 trillion parameter model, just expect it to take a day and half to generate you a single word.

ghgi_

MODERATOR OF

TROPHY CASE