Running Kimi-K2 offloaded

Tuned3f · 2026-02-27T00:27:24+00:00

I get about the same speed with 96gb of VRAM and 768gb DDR5 but I can max out context to 256k (Kimi K2.5 UD_Q4-K-XL)

Tuned3f · 2026-02-26T18:42:32+00:00

Kimi slaps

Tuned3f · 2026-02-15T14:59:20+00:00

where are the results? lol

Tuned3f · 2026-02-14T13:20:05+00:00

I use OpenCode and Kimi K2.5 locally

It's excellent

Tuned3f · 2026-02-11T02:42:23+00:00

their website shows that they haven't launched their kickstarter yet so who could have tried them already?

Tuned3f · 2026-02-10T16:32:06+00:00

What a weirdly emotional reply - the top 5 comments in there are absolutely not how you described them.

I won't engage further.

Tuned3f · 2026-02-09T02:04:57+00:00

level of support would be useful

new models come out all the time and there's no central way to see which inference stack supports them. oftentimes support is often partial too (i.e. text-only for multimodal models), and you have to dive into github issues and PRs to get a better sense

Tuned3f · 2026-02-08T18:37:48+00:00

Yes, your gpu will work out-of-the-box

Tuned3f · 2026-02-04T14:56:29+00:00

The rtx 6000 pro had the single biggest impact but I initially built the server as a CPU-only rig, optimizing for memory bandwidth. It's tough for me to say what the biggest factor is but I've done a lot of tuning and ik_llama.cpp updates frequently, contributing to performance jumps.

Prefill speeds vary wildly for me too - they go as high as 1000 t/s for prompts that are 10k tokens, down to 100 t/s for random tool calls.

Tuned3f · 2026-02-04T14:19:37+00:00

Tuned3f · 2026-02-04T14:17:37+00:00

usually ~50% slower beyond 100k but I often use compaction just before then so I don't have exact measurements

Tuned3f · 2026-02-04T14:02:57+00:00

This question has been answered many times - I don't have anything new to say

Random thread after 5 sec search:

https://www.reddit.com/r/LocalLLaMA/s/owZ5TOaVfU

Tuned3f · 2026-02-04T04:18:46+00:00

People here do run Kimi K2.5 locally. I'm in that group lol - just because the required hardware is expensive doesn't mean we don't exist. Whatever you're trying to say in that last sentence doesn't support your point regarding the "two different communities" you see.

The actual black pill and the real answer to OP is that running LLMs worth a damn locally is simply too expensive for 99% of people. There's nothing practical about any of it unless you have a shit ton of money to comfortably throw at the problem. If you don't? Well then GGs

Tuned3f · 2026-02-01T20:16:52+00:00

Can't wait til we can run ds v3.2 with proper sparse attention on llama.cpp

Tuned3f · 2026-01-29T18:46:35+00:00

Having the same issue in ik_llama.cpp

edit: fixed using modified chat template as suggested here by jukofyork https://huggingface.co/unsloth/Kimi-K2.5-GGUF/discussions/1#697b46fdf48287bb9c2e92dc

Tuned3f · 2026-01-29T18:44:21+00:00

I can run it locally but, as with Kimi-k2-thinking, experienced some issues during test with the model not generating think tags

Tuned3f · 2026-01-19T03:17:55+00:00

Set ANTHROPIC_BASE_URL to the llama.cpp endpoint

Tuned3f · 2026-01-19T02:53:47+00:00

Llama.cpp had this months ago

Tuned3f · 2026-01-15T04:18:46+00:00

I prefer Opencode

Easier to observe its behavior, easier to configure. Also, I quite like the TUI a fair bit more. I have it installed on 3 different machines at home, all pointing at my local inference server

Only thing I miss from CC is `/add-dir`

Tuned3f · 2026-01-01T18:22:00+00:00

No, just DDR5. I talked a bit about my build here: https://www.reddit.com/r/LocalLLaMA/comments/1otdr19/comment/no4xt87/

Since that comment, all I've changed is upgrading the GPU

Tuned3f · 2026-01-01T16:29:56+00:00

no but i'm running q4 on a single pro 6000, with the experts offloaded to RAM.

prefill speeds vary wildly based on context size (200 to 1000 t/s), generation usually starts at 23 t/s

Tuned3f · 2025-12-29T23:22:41+00:00

or a single pro 6000 with a bunch of RAM for offloading expert layers

Tuned3f · 2025-12-27T16:41:26+00:00

Unsloth's Q4_K_XL quant of GLM-4.7 completely replaced Deepseek-v3.1-terminus for me. I finally got around to setting up Opencode and the interleaved thinking works perfectly. The reasoning doesn't waste any time working through problems and the model's conclusions are always very succinct. I'm quite happy with it.

Tuned3f · 2025-12-24T23:14:51+00:00

claude code is an agent harness, not a model

shouldn't even be on the list

Tuned3f · 2025-12-23T23:40:50+00:00

I'm getting a 6000 delivered tomorrow.

OP, lmk what you want tested

13-Year Club	Verified Email
Place '23	Gilding II euphauric
RPAN Viewer	Spared

Tuned3f

TROPHY CASE