Even though they keep getting caught when they cheat on benchmarks, Android vendors keep doing it. Prominent handset makers that created some of the best iPhone rivals around cheated in benchmark tests in previous years, including Samsung and OnePlus, and they were all caught. Huawei now joins the sad list after several of its handsets, including the P20 Pro, were discovered behaving differently when a benchmark app was detected.
It all started with AnandTech, which first discovered the behavior and then talked to Huawei at IFA 2018 about their findings. Huawei acknowledged the discovery, stating that they’re doing it to compete against rivals from China that are also cheating in similar tests.
Huawei then provided a comment to Android Authority in which it explained its behavior:
Huawei always prioritizes the user experience rather than pursuing high benchmark scores – especially since there isn’t a direct connection between smartphone benchmarks and user experiences. Huawei smartphones use advanced technologies such as AI to optimize the performance of hardware, including the CPU, GPU, and NPU.
When someone launches a photography app or plays a graphically-intensive game, Huawei’s intelligent software creates a smooth and stable user experience by applying the full capabilities of the hardware, while simultaneously managing the device’s temperature and power efficiency. For applications that aren’t as power intensive like browsing the web, it will only allocate the resources necessary to deliver the performance that’s needed.
In normal benchmarking scenarios, once Huawei’s software recognizes a benchmarking application, it intelligently adapts to “Performance Mode” and delivers optimum performance. Huawei is planning to provide users with access to “Performance Mode” so they can use the maximum power of their device when they need to.
Huawei – as the industry leader – is willing to work with partners to find the best benchmarking standards that can accurately evaluate the user experience.
UL, the creators of the famous 3DMark test, performed their own research and decided to delist four Huawei phones that have been caught cheating in 3DMark, including the P20 Pro and P20, but also the Nova 3 and Honor Play.
The company tested all phones using the public version of 3DMark that’s available on Google Play, as well as an internal one that Huawei didn’t have access to it:
We found that the scores from the public 3DMark app were up to 47% higher than the scores from the private app, even though the tests are identical.
This is proof that Huawei’s software is actively bumping up performance when detecting benchmark apps, but also that the AI features Huawei promotes don’t always work, because they failed to increase performance in a similar way when the phones were tested using the internal builds.
It’s bizarre to see Huawei engage in benchmark testing and hope it won’t get caught. The findings are all the more annoying because Huawei just unveiled its next-gen processor, the 7nm Kirin that will compete against Apple’s A12 and Qualcomm’s Snapdragon 855. And the company used benchmark scores in its presentation to highlight the Kirin 980’s advantages over the current Snapdragon 845. When the Mate 20 Pro comes out next month, it’s likely the phone will be put through thorough benchmarks with testers looking for any signs of cheating.
For the second time in a matter of months, Huawei has had to defend itself against shady practices. A few weeks ago, Huawei was just caught using a DSLR camera for a commercial that implied the selfies in the ad were taken with a new Huawei handset.
Even though benchmark tests aren’t entirely reflective of the real-life performance of a phone, misleading customers isn’t the way to go, especially for a company looking to become the number one smartphone vendor in the coming years.