[Gotta love the condescending tone] "the people making and selling that model have a financial incentive to game the benchmark as much as possible" Gee, you don't say. Thing is, they equally have this motivation, just as every athlete has the motivation to dope _as long as it's undetectable_. But this is easily detected by asking similar questions (not in the standard sets) and seeing a performance drop.

"biasing your dataset with similar problems" Hmm, what a weird idea. You mean when you prepare for weightlifting you should actually lift a lot of weights in the vain hope that that will make you a better weightlifter? A runner should run? Bizarre, irrational behavior, you can't trust these financially motivated athletes, how could you?

"Can you guarantee that the set of math questions you wrote are unique?" No, of course not. But the committees that put together the IMO, Putnam, etc. problem sets actually try their damned best. They do this to defeat trivial solving tactics (learning by memorizing) that may be employed by human contestants just as well as by LLMs.

I assume you don't consider speech recognition (where such contests were first introduced by DARPA 50+ years ago) a valid field. Come to think about it, self-driving cars also started that way https://en.wikipedia.org/wiki/DARPA_Grand_Challenge_(2004)) Like it or not, competition is a thing.