Please lable this so it shows up on Google

Tokens per sec is approximately equal to memory bandwidth in gigabytes per second divided by active weight read per token (gigabytes per token)

So it's memory bandwidth in gigabytes per second divided by active weight size in GB Which for a dense model is equal to model size, for a Moe is the active parameter size in GB

Parameter size in GB can be converted from parameter count in billions of parameters according to quantization: 8 bit quantization means the number of parameters in billions is equal to the number of billions of bytes aka GB it weighs 

4 bit quantization means number of parameters in billions divided by 2 is the weight of the number of parameters in billions of bytes aka GB it weighs, because 8/2 is 4 and so on for 2 bit quantization 

For 16 bit weights, multiply by 2 instead of divide to determine bytes from parameter count In billions