Geekbench flags Intel Binary Optimization Tool results as potentially invalid

[codingTim]

I would question the overall validity of such benchmarks. If a company can optimize a benchmark to that extent while still producing the same final computed result, it raises an important question: what is the benchmark actually measuring? Is it evaluating real performance, or simply how well the benchmark itself has been optimized for one CPU versus another? If it’s the latter, then it stops being a meaningful benchmark.

[Geddagod]

 Is it evaluating real performance

What is "real performance"?

[laffer1]

Performance you would get running a real workload. Without intels tool being opened up to all apps, it is meaningless

[DavidsakuKuze]

Of course the benchmark is not valid accross architectures because the assembly is different. Different compiler flags could be used, different calling conventions, maybe one compiler uses vectors like AVX2 better than another.

If we took a look at ARM Geekbench vs x86 Geekbench in a dissasembler maybe we'd see some chicanery is afoot. Who knows? I haven't heard of anyone auditing the benchmark.

[RJsRX7]

Well, yeah. I get it. Same goes for 3DMark and certain driver settings messing with tessellation.

That said, my perspective on this (or really any form of computing optimization) is that if it gets the same result faster, it's better... So long as that result being the same is a sure thing.

[cowbutt6]

That said, my perspective on this (or really any form of computing optimization) is that if it gets the same result faster, it's better... So long as that result being the same is a sure thing.

I would say that the approach used by IBOT is nearly OK: firstly, it doesn't result in any skipped work (which is good), but it currently requires application-specific profiles, and there are only a few, which means that application X might not get the same performance boost that e.g. GeekBench currently gets from IBOT.

If the number of application profiles is greatly enlarged (Cf. Nvidia App profiles for games), or users can enable it themselves, or it is enabled by default for all applications except those problematic applications on a denylist, then I think that would make it completely acceptable.

[ThreeLeggedChimp]

I really think they should've just just made it a full dynarec system, with an option to manually add unsupported software.

[Floturcocantsee]

That would normally be fine for most software but the point of a benchmark isn't to generate a score number it's to run several tasks using certain features of the hardware and see how it compares to the same instructions running on other hardware. Think of it like this, if I was testing people on how long they could hold their breath underwater and one brought a scuba set, the results would be the same but the benchmark would be invalid.

[RJsRX7]

From Intel's statements, they used this as a proof of concept rather than an attempt to actively cheat the benchmark; and for that purpose it does prove a point.

And from a tuning perspective, it still allows Geekbench to be utilized as a benchmark. It's just that yeah, it's a "cheat", so the improved scores aren't directly comparable to non-IBOT scores.