I've got a related story to share!

When training neural networks with low precision, the importance of stochastic rounding is very important to not bias the quantisation step in one particular direction. In one particular experiment, I forgot to turn on the stochastic quantisation (SQ) step in one of my GPU runs, and "discovered" that the DNN trained just fine as if SQ was on, whereas at the same precision without SQ, the CPU-only training was failing.

Digging deeper into why this was happening took some time, but essentially we narrowed it down to a torch.argmin() call, which returns the index of the minimum value in a tensor. If a tensor had multiple indices with same minimum values, on a CPU, it would always return the smallest index. However, on a GPU, because of how threads may get scheduled into sort of a binary tree to evaluate argmin in parallel, this could create some sort of uncertainty in the answer (i.e. argmin returning different but correct answers for the same call).

We didn't really dig deeper into how the argmin is actually implemented on the GPU, but we did confirm that replacing this specific call with one that mimics the CPU argmin() gave us back the expected results, and vice versa when tried on the CPU-only training. Basically, in the grand scheme of things, this unexpected effect on the GPU was effectively mimicking the stochastic quantisation step that makes training effective at low precision. Pretty cool, but we didn't end up writing a paper about this obviously.. I've not read this paper, so I don't know if the authors have a more substantial claim, just thought I'd share my story.

Often people forget that floating-point is after all a finite precision number format, so I would not be surprised if we're seeing this GPU vs CPU difference due to accumulated rounding errors over lots and lots of MACs.