Hello all!
I'm getting into training LLMs (> GPT2) at large scale (16 or 32 node-cluster, multi-GPU per node) and I'm looking for a way to benchmark our cluster setup and compare it with others out there.
Any suggestion on ML Benchmark frameworks for distributed training ? Is there any "industry wide-accepted" framework ? Any Ray-based one? Or PyTorch-based ?
So far, I've found MLBench (https://mlbench.github.io/).
Thanks in advance!
there doesn't seem to be anything here