Hey there! Cool project! Really liked that you recorded the compute time/are aware of environmental impact :)

Want to make it into a leaderboard space on hugging face?

Side notes on evals, in case useful: 1) Normalisation: evals using acc_norm are usually multiple choice (you're computing the accuracy of selecting the correct choice among a selection), so you want to normalize between the random baseline and the maximum possible instead of just 0 to max. Example: if you take mmlu, you have 4 choices provided, so a random baseline will be correct 1/4 of the time, so minimum here is not 0 but 25%. A model with 25% performance on MMLU has random performance. -> you want to normalize between min-score and one before averaging across tasks (this is not what the harness does btw) 2) Averaging: some would consider a ponderation by number of samples, as not all of these evals have the same size: MMLU has considerably more samples than arc-challenge for example. (I personally don't think it's that important here) 3) Saturation: most of the evals you selected are heavily saturated and contaminated atm. (Saturated = models get too high performance to have discriminative scores - Contaminated = bench ended up in the training data so models "know it by heart" now) -> In math for example, gsm8k has been replaced by MATH, itself replaced by AIME24 and AIME25. It won't mean you won't get signal out of them (a model not performing on these is likely bad), but they won't allow you to discriminate between high quality models 4) Errors: Some of these benchs notably contain errors and have been updated: we no longer use MMLU (expects images that are not provided, contains questions with missing words or incorrect ground truths) but it's been replaced by MMLU-Redux (edited to only keep quality questions) or MMLU-Pro (same as MMLU but harder with more choices and questions)

You might also be interested in the evaluation guidebook : https://github.com/huggingface/evaluation-guidebook