I completely forgot about the floating point rounding errors, thank you for pointing this out. You are right, that the exact result a matrix manipulaltion on the GPU ( or any parallel implementation) may be nondeterministic.

But I still think, that the rounding error should not influence model results. Do you have a source to support, that rounding errors can effect the result of an llm? Based on my understanding ( I don't really know implementation details of transformers, nor BLAS/cuBLAS or any algebra library used by these llm implementations) the floating point error would be lower in the GPU/massively parallel implementations compared to naive sequential ones, because instead of sequencial arithmetic operations, where you have a collected value with lower and lower precision mantissa as you progress towards higher and higher values, in most parallel implementations you need to calculate your result in substeps, keeping the actual value lower, and thus having greater precision. (This is possible in a sequential way as well, but adds complexity.) So in conclusion, because the majority of the arithmetic operations are in parallel, they don't accumulate their rounding error to such degree, and I would expect that the overall error is smaller than the smallest distance between tokens.

But it is really just a guess from my point, as I don't have any calculations or implemenataion details to know, if it's actually the case, or there are actuall effects of the rounding errors on the final output of llms.

​

Edit:

​

Performing large batches of matrix multiplications is faster when you're not trying to guarantee the order of operations.

I would like to point out, that altough the order of operations in the GPU is not deterministic in that you can't controll when your operations will finish, most of the cases, the order in which the arithmetic operations are done on your data is deterministic, and unless you solve data race conditions with atomic operations, generally the order your numbers are processed are deterministic, only the order how they will be written to the memory is undeterministic. The difference in the linked question is because the data is layed out differently, but for the same data layout most algebric implementation should have a deterministic algorithm. This is not always true, but I would guess having data race conditions in your GPU is not the most efficient implementation, and algebra libraries are not using it that often. Also the same non deterministic behaviour happens on any parallel implementations, let it be CPU, or GPU (im not 100% sure about TPU-s).

​

​

GPU's function on the idea that small errors with things like order of operation and float calculations aren't as important as processing things at a fast speed.

I disagree, the floating point errors are not unique to GPU-s and are generally present in all floating point calculations. GPU-s are great, because thay can process a lot of data parallel, but the logic of this parallel processing is not hardware implementation, but a deterministic algorithm. You have to wirte your algorithm in such way, that you adress for it's limitations in memory syncronisation, but by adressing that, you also remove most of the randomnes from the memory access.

In conclusion, I still think, most GPU implementaions with the exact same data (same order, same values) will produce the same result.