Huawei Ascend P7 Banished From 3DMark Ice Storm Rankings For Alleged Cheating
In Futuremark's world, you either play by the rules or you're banished from the benchmarking kingdom. It's happened before -- on more than one occasion, actually -- and it just happened again, this time to Chinese handset maker Huawei and its Ascend P7 smartphone. Futuremark discovered that the Ascend P7 was using a trick to artificially inflate its 3DMark score, so the handset's been removed from the 3DMark performance rankings and now appears delisted and scoreless in the company's list of Best Smartphones and Tablets.
One of Futuremark's rules is that "the platform may not detect the launch of the benchmark executable [and] the platform must not...modify the usual functioning of the platform based on the detection of the benchmark." Otherwise, it opens the door to benchmark optimizations, which in turn creates an uneven playing field and doesn't offer up a true representation of real-world performance.
Futuremark cites Anandtech's detective work as further proof of benchmarking shenanigans. That detective work includes a benchmark comparison of the Ascend P7 running the Play Store version of 3DMark that's available to everyone and a renamed (but otherwise identical) version not available to the public. In the official release, the Ascend P7 was able to detect the benchmark and apply optimizations, resulting in a score of 7,462. When running the renamed version, the handset scored 5,816, which is 28 percent slower.
"This clearly shows that the Ascend P7 is detecting 3DMark by name, and not by workload. If the 'CPU configuration is adjusted dynamically according to the workload in different scenarios,' as claimed by Huawei in their response to Anandtech, then the score from the renamed version of 3DMark would be identical to the public version," Futuremark says.
Huawei isn't alone in being called out for shady benchmark tactics. Futuremark previously delisted several Samsung and HTC handsets, including the HTC One M8. Some of those phones have been added back after the manufacturer made some changes to comply with the rules, such as the Samsung Galaxy S4 and Note 3.
One of Futuremark's rules is that "the platform may not detect the launch of the benchmark executable [and] the platform must not...modify the usual functioning of the platform based on the detection of the benchmark." Otherwise, it opens the door to benchmark optimizations, which in turn creates an uneven playing field and doesn't offer up a true representation of real-world performance.
Futuremark cites Anandtech's detective work as further proof of benchmarking shenanigans. That detective work includes a benchmark comparison of the Ascend P7 running the Play Store version of 3DMark that's available to everyone and a renamed (but otherwise identical) version not available to the public. In the official release, the Ascend P7 was able to detect the benchmark and apply optimizations, resulting in a score of 7,462. When running the renamed version, the handset scored 5,816, which is 28 percent slower.
"This clearly shows that the Ascend P7 is detecting 3DMark by name, and not by workload. If the 'CPU configuration is adjusted dynamically according to the workload in different scenarios,' as claimed by Huawei in their response to Anandtech, then the score from the renamed version of 3DMark would be identical to the public version," Futuremark says.
Huawei isn't alone in being called out for shady benchmark tactics. Futuremark previously delisted several Samsung and HTC handsets, including the HTC One M8. Some of those phones have been added back after the manufacturer made some changes to comply with the rules, such as the Samsung Galaxy S4 and Note 3.
