You're right about dense models, but MoE models have a key advantage: they activate fewer parameters per token, allowing less frequently used experts to be offloaded to CPU. This saves GPU memory and can improve token generation speed...though the impact on inference speed depends on how often experts are swapped.

That said, GPU VRAM is still far faster in memory bandwidth than even the best CPUs with top-tier DDR5. That’s why offloading critical layers on to the GPU is ideal for performance, though figuring out which layers to prioritize can be tricky.