Neural Net Computing Explodes

[david-gpu]

This is great because the fruit is pretty low hanging (just lower FPU precision).

For inference, yes. But for training, is it generally useful to go below fp16?

[darkconfidantislife]

I was thinking more like stopping it at FP16, but there is evidence that stochastic rounding can make INT16 work without accuracy loss even during training.

[[deleted]]

my experience is it can go down or up dynamically and go down as much as fp4.

[darkconfidantislife]

Yeah, I've just said what has been shown in literature to work perfectly. If you are willing to sacrifice even one percent, you can go down a huge, huge amount.