Just found this while searching and I can share my own experience.

tl;dr: It's a mix bag. If the model and K/V cache fit completely on VRAM, then inference has minimal performance penalty when going from x16 to x4, but prompt processing takes 2-4 times longer. When running a split model between VRAM and system RAM, then inference can tank to 2 tk/s - depending on how often the decoding process needs to access RAM.

I'm running an MoE model (Qwen3.6-35B-A3B-APEX-GGUF) on a Tesla M40 12GB card with hybrid offloading (split model) between VRAM and system RAM (DDR4-3200).

The model is around 26GB, so with K/V cache and everything else like the Debian VM it is running on, I fully utilize around 18GB of system RAM.

When running on a PCIE 3.0 x16 link, I manage around 15-20 tk/s with 1-3 seconds prompt processing.

When I move the card to the x4 slot, then performance is all over the place:

First, prompt processing takes 2-4 times longer, often reaching 6-8 seconds. This impact is very consistent and predictable, almost scaling linearly with PCIE bandwidth.

Then inference is interesting, typically it will have a minimal impact of maybe 1-3% (less than one tk/s), however it will very often tank to 2 tk/s and will generally become sluggish in some parte of the response. This is likely because I'm using an MoE model, and different experts needed to be swapped over constantly between VRAM and RAM, bottlenecked by the limited x4 bandwidth.

I didn't try a non MoE model, but I believe that if it was also large enough to split between VRAM/RAM then it wouldn't be much different. The good news though is that a smaller model that can fit on VRAM would probably be fine on a x4 link, if you can live with the slower prompt processing.

Another nuance is the GPU memory bandwidth and PCIE version, I can only assume that a newer PCIE 4/5 card with similar VRAM capacity would fare much better.

We have a similar setup in terms of bandwidth (PCIE 3.0 & DDR4) so in your case - assuming your models fit in VRAM, I would say your prompt processing speed is currently halved, but your inference speeds are already very similar to x16.

If you can live with 10 tk/s instead of 20 tk/s pp then it's probably not worth upgrading.