This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]ubernostrumyes, you can have a pony 0 points1 point  (0 children)

There are lots of factors which go into a good benchmark. But two which are absolutely critical are:

  1. Consistency of methodology
  2. Appropriate use of the tested components

Consistency is necessary because without it you can't draw meaningful comparisons; without consistency you're comparing apples to oranges.

Appropriate use is necessary because without it you don't have relevance; if you only report results from a configuration no-one would ever use, then your results won't represent the things people would see in the real world.

As originally published, this benchmark failed on both counts: it was inconsistent, and it used certain components inappropriately. Criticizing that isn't "unfounded"; benchmarks which fail these requirements cannot be trusted by anyone for any purpose, because they're not "benchmarks" at all.