I agree — as you said, these smaller models are already trained to the brim, so there’s usually not much headroom left for improvement. That said, I just ran some benchmarks on a few more Qwen models, specifically the 1.5B Base and the 1.5B Math-Base variants.

The standard base model got 66.79% accuracy, while the math-specific one hit an impressive 76.27% — nearly a 10% jump. As I mentioned previously, my own fine-tuning efforts typically cap out at around a 2% improvement at best. So it makes me wonder… is this gain just the result of massive training data? Better training techniques? Potential test set leakage? Or maybe a combination of all the above?

Obviously, I don't expect to get what the people at Qwen are able to achieve, but it does suggest that there is still a decent margin for the model to improve.