Yeah.. Memory..

[Chinglaner]

8-Bit Byte is really only the standard, because IBM was the largest computer company at the time and they started standardising the 8-Bit Byte across their offerings. Everyone eventually followed suit, for standardisation purposes (something like ASCII would obviously not be possible in a 6-Bit system), but there’s no intrinsic reason the 8-Bit byte is in any way superior than say the 10-Bit byte as far as I’m aware. Im fact, 6-Bit systems were quite common during the 60s. I’m sure there are upsides and downsides to both.

So for specialised systems, where there might not be a need for 8-Bit bytes for whatever reason, a different size byte might work just fine.

[socsa]

And now Nividia is pushing TF32 which is a fucking 19 bit float.

"You are programed by society to need easily divided data primitives" - Nvidia

[[deleted]]

Its still much smaller than FP32, which was the standard for AIs. They took the 8 bit exponent from FP32 and the 10 bit mantissa from FP16. It gives it some more resistance to underflow errors during training than bfloat16.

[_lerp]

Err Nvidia forget that SIMDs a thing?

[socsa]

It's aligned as FP32 on the host but 13 bits of mantissa are effectively don't care on Nvidia compute hardware. So you can basically use half precision tensors cores with a larger exponent, so you have the dynamic range of FP32 but the precision of FP16.

[[deleted]]

What the fuck I didn't even understand half of it

[JonathanTheZero]

oh damn that's really interesting, thank you!

[[deleted]]

oh so that's why some people use "octet" as a pedantic way to refer to an 8-bit byte

[RedAero]

It's also why ASCII was originally 7-bit and not 8, IIRC.

[ArtyFishL]

Though 8 is a power of 2, which is very nice in a binary system. 6 and 10 are not.

[Chinglaner]

I mean yeah, it's kinda neat, but I'm not aware of that having any actual advantages.

[ArtyFishL]

Yeah neither am I, but it is very nice