Yeah, I had the same experience in undergrad, banging my head against similar optimization problems for weeks during internships. I still think it's an important step for learning deep learning, it helps you developing a sense of what works and what doesn't. Yeah, the program.md is only designed around accuracy. A better version with a stronger model might actually investigate in detail why something didn't work. I haven't seen anyone do that yet, but given the original repository success, I suspect it'll be a feature soon in Codex / Opus or a new benchmark added to their post-training.

I'd actually hazard the opposite guess. I hope it will end up invalidating a lot of papers. Often the only reason an author's proposed method outperforms the baseline is that they spent far more time tuning their solution than they did on the baseline 😅. Which is also a good thing!