Monday, December 2, 2013

Samsung and HTC have been found to tamper with their devices’ benchmark numbers. Many are appalled by their practice. However, if Samsung and HTC were making cars, instead of smartphones and tablets, would they still be labelled as cheaters?

In the past, to be a petrol  head was to be familiar with wrenches, hydraulic jacks and the smell of oil on one's person. Nowadays thanks to the magic of video games like the Gran Turismo, Forza, or the Need for Speed franchise, people can be car experts without ever stepping into a grimy garage. These games often provide detailed spec sheets of the cars they feature including things like horsepower, torque, as well as 0-60 times. Modern day arm chair car pundits can recall horsepower figures of cars like the BMW M3, C63 AMG, or Audi RS4 out of the top of their heads without ever experiencing the terror of thinking whether the straps on a dynamometer (dyno, for short) would be strong enough to hold the car and stop one's car from getting launched into the garage wall while trying to get a proper "dyno run".

One thing that some people often don't realize is the fact that most of these manufacturer benchmark numbers were obtained under highly controlled circumstances. Take horsepower for example. Let's say you have a current model BMW M3. Its spec sheet tells you that the car's engine will produce 414 horsepower at 8300 RPM. However, if you actually take your car to a dyno, you will realize that no matter how hard you try, it is impossible for you to get the kind of horsepower numbers indicated on your car's spec sheet from a showroom stock car.
The reason for this discrepancy is that the horsepower number most car manufacturers quote on their spec sheet was not obtained by running the car on a rolling road to measure the horsepower output from the cars' driven wheels. In most cases, these numbers were obtained by putting the power train (engines, exhausts and auxiliary components) on a rig, in a lab, and then measuring the peak horsepower coming out of the engine's crankshaft. Thus, horsepower ratings on spec sheets will always be higher than what the consumers can ever get from running their cars on a dyno. Unless specifically stated that the horsepower figure consumers see was "wheel horsepower", every car manufacturer is, literally rigging their horsepower benchmark numbers and they have been doing this for decades.

AMG testing the 6.2l V8 Mercedes-Benz SLS AMG engine.

In the past few weeks the blogosphere has been lit up by news that some Android manufacturers have been found to "rig"  the benchmark numbers of some of their popular devices. Last week news broke out that a popular Benchmarking suite, 3DMark has delisted a number of popular devices from Samsung and HTC because said devices were found to use an exclusive high performance settings while running the benchmark app to allow them to produce numbers that are higher than what they could have achieved if they were running on their normal settings. Needless to say, we have seen many tech commentators surprised and  downright appalled by these practices.

However, what Samsung and HTC have done, while unfair to other manufacturers, is not exactly new. Car companies have been doing the same thing for ages. Samsung and HTC rigged their benchmark tests the same way car manufacturers rigged their engines to come up with their horsepower ratings. The practice may be new in the mobile space, but it is not exactly  unprecedented in other industries.
It is true that this "rigging" make Samsung's and HTC's benchmark numbers not representative to their device's actual performance, but they are not anymore misleading as the horsepower figures that car manufacturers put in their promotion materials. If Samsung and HTC are branded guilty for providing misleading benchmark numbers, then they are merely as guilty as Mercedes-Benz, BMW, Audi, Toyota, Nissan, or (put your favorite car manufacturer here).

The auto industry has been around for more than  a century, yet the mobile device industry has only been around for less than a decade. In time, manufacturers will come up with a standardized way of providing benchmark/performance figures. Whether or not they will eventually reach the same consensus as their automotive counterparts, with everyone literally rigging their benchmark numbers unless otherwise specified, remains to be seen.
Furthermore, serious car tuners know that horsepower numbers are only useful when used as a tuning aid. In performance tuning, these numbers act as benchmarks (thus, the name) to measure the success of subsequent tuning approaches. These numbers mean little in comparing between measurements taken with different instruments, much less between different cars.
In short, unless you’re doing this, benchmark numbers don’t mean much.

To see the proof of this, one only need to observe this year's Formula 1 season. The Formula 1 series is sanctioned by a governing body called the FIA. This organization provides strict regulations or "formulas" for the sports' competitors to follow. Although this governing body never actually disclose the exact power output of Formula 1 cars, most analysts agree that judging from the regulations, this year's Formula 1 cars produce around 750 horsepower. One might assume that since all of the cars produce similar horsepower and have near identical specs, the competition must be pretty tight.

However, anyone who followed the 2013 Formula 1 season will tell you that that couldn't be further from the truth. One driver won 13 out of the 19 races available throughout the season. The same driver won 9 out of the last 9 races. Needless to say, this driver have also won 4 out of the last 4 world championship titles.
This demonstrates the importance (or lack of it) of benchmark numbers to real world performance. That is a fact that applies to cars as well as mobile devices.

In an article posted on October 9th, Anand Lal Shimpy and Brian Klug of Anandtech mentioned,

"The hilarious part of all of this is we’re still talking about small gains in performance. The impact on our CPU tests is 0 - 5%, and somewhere south of 10% on our GPU benchmarks as far as we can tell. I can't stress enough that it would be far less painful for the OEMs to just stop this nonsense and instead demand better performance/power efficiency from their silicon vendors. Whether the OEMs choose to change or not however, we’ve seen how this story ends. We’re very much in the mid-1990s PC era in terms of mobile benchmarks."

So what can we do as end users? 

The first thing we can do is realizing that benchmarks are crude estimates of a product's real world performance and, just like the case with cars, they are only actually useful as tuning aids for when we conduct modifications to increase the performance of our devices.

For those of us who are not into performance modding, at its very best, benchmark numbers provide very rough performance estimates that we can use to attempt to group different devices by their estimated performances. This can help us in making our next purchase by reducing the number of candidate devices . It should not, however, be used as a primary variable to decide which of those candidates to eventually buy. To make our final decision, it is best to do what car buyers have been doing for decades, which is giving the cars, or in our case mobile devices, a proper test drive. 

Next time you're comparing benchmark scores, keep in mind that a 750 horsepower Pagani Zonda F is not 25% slower than a 1000 horsepower Bugatti Veyron. In fact, on a number of occasions, the lesser powered Zonda is actually the faster car.

Images and videos courtesy of: