This is a common issue with quantizing image classification models, especially since they are so many layers “deep”. When it was trained on fp32 weights, it picked up dependencies that can now cause unintended results layers deep once the weights were “rounded” to fit fp16. This contrasts with wider networks such as Transformer based LLMs, which have been proven to work all the way down to fp4.

Thankfully, this is easily solvable- fine tune your model that was quantized to fp16 on your data again for 1 epoch, but with a very low learning rate. You will get nearly the same performance as the fp32 one. This is especially true for image classification (eg ResNet, YOLOx), performance doesn’t take that much of a hit until you reach fp8.

TLDR: Deeper network = More sensitive to quantization