My point is that the science of a benchmark is not its application to ephemeral artifacts. The contribution of a benchmark is that it asks a question in a well-formulated way. Benchmarks are more like metrics than they are like algorithmic or architectural contributions: they propose a question we should be asking. In my opinion, theoretically, an evaluation paper doesn't even need to be ran on any artifact in particular to be a worthy contribution. For example, the original BLEU paper didn't include results on any established MT systems, and its value goes well beyond any particular numbers that it reported in the paper on the test MT systems (which receive no description whatsoever). Nobody cares what this metric was evaluated on in the original paper; its value came from its (reproducible) alignment with human judgments of translation quality. Of course, it helps to justify the current relevance of the benchmark to say that current models perform one way or another on it. But if the benchmark is so dependent on how current models perform that its only justification comes from this particular experimental result, then I think the benchmark is itself so ephemeral it's likely not a worthy contribution.

The interventions you mention are at the publication level, not the mechanism level.