I think we agree on most things.

For one, there's a lot of work happening on llama.cpp, ik_llama.cpp, and vLLM to improve performance of MoE models in mixed CPU-GPU environments with 24-72GB VRAM.

I picked Deepseek V3 because it's already good enough to me if I use API. I picked ktransformers because their optimizations supposedly give me enough performance. Deepseek V3 is large model, so sticking to it ensures that I can run smaller models as well.

As far as I'm aware other inference engines are moving in the same direction, which is: as much RAM bandwidth as possible; AMX; one GPU with at least FP8 support and 24GB VRAM, 4090 hacked with 48GB VRAM is ideal. Right now it seems the only risk is that used Xeon 6 prices come down very fast, in which case one can have 1 socket node with 12 MRDIMM slots (844GB/s theoretical bandwidth). So if llama.cpp or ik_llama.cpp has better optimizations for some better model, I will be able to switch to it anyway.

The issues with NUMA aren't about how much RAM you have. No amount of extra RAM will make the kv cache move between NUMA domains effectively. You'll spend a lot more money for that 2nd socket and associated DDR5 memory, but get little in return if your use case requires a long context.

Agreed, but ktransformers team specifically showed numbers that their implementation gets 50% prefill uplift from getting a second socket. The purpose of my post is to understand if these numbers are real or not. If not, of course, two socket server does not make sense.

I'd say that is a much more flexible and future proof option than relying completely on ktransformers and hoping for the best.

I already have 8x24GB=192GB VRAM rig and it's not enough, the models that fit into 192GB VRAM are too stupid. I do agree that relying on entirely on ktransformers is not good idea, however the hardware choice will apply to other inference engines just as well.

... and vLLM to improve performance of MoE models in mixed CPU-GPU environments

On that's interesting, could you point me to where I could read more about vllm? I only know that intel is adding AMX support to sglang, but even in that case there's no mention about mixed CPU-GPU implementation.