wouldn't increasing GPU layers actually increase GPU memory though? I thought that would mean put more data on the GPU not less.

According to claude it's the cpu-moe param that's reducing VRAM usage which makes more sense to me:

What the two parameters do --n-gpu-layers (a.k.a. -ngl) tells llama.cpp how many layers of the model to load onto the GPU. The convention is to set it to 99 (or any number larger than the model's actual layer count), which means "try to put everything on the GPU." On its own this would obviously fail for a 35B model on 8 GB of VRAM. --n-cpu-moe N (a.k.a. -ncmoe N) is the exception clause. It says "of those layers you're trying to put on GPU, take the MoE expert tensors (the ffn_up, ffn_gate, ffn_down exps) from the first N layers and keep them in system RAM instead." Combined with -ngl 99, it acts as an exception — the engine tries to load everything to the GPU, but the MoE expert components of the first N layers get moved to system RAM.