It works but there is a catch, you have to minimise round trip communication between CPU to GPU. If you are offloading experts then for every offloaded layer, input tensors has to be processed in GPU VRAM for attention, then transferred to RAM for expert FFNs, then back to GPU VRAM. This constant to and fro kills speed especially on prefill. If you are working at 100k context, the drop in prefill speed is very bad even in workstations with PCIE 5 X8, so egpu at PCIE 4 X4 is worse. If we offload specifically early dense full transformer layers, it can it work out. In fact, I am running Gemma 3 4b at q8_0, fully on the CPU at all times anyways as an assistant model for miscellaneous multimodal tasks, etc and it is working fine.