The problem with Geekbench benchmarking and Android

There has been a lot of news lately about how slow the androids phone are compared to the latest iPhone 5. The main benchmarking suite I have seen quoted is Geekbench. The articles quoted the iPhone 5 at 1601, and the fasted android, the Samsung Galaxy S3, at 1560. Making all the apple owners jump up in joy.

To be honest I never heard of Geekbench until I saw articles of how much faster iOS is compared to android. This is hardly surprising, because if you do a search for benchmarking on Google Play, Geekbench 2 (the current version) does not appear until the third page of results. (This was search result at the time of writing).

Also this is a paid application. This is important because of the way Geekbench works. It posts your results to a central server, and the results are accumulated and can be seen at http://browser.primatelabs.com. Paid applications on Android are not that popular, especially when there are lots of equivalent free ones. So the average android user who wants to run some benchmarks will inevitably instal one of the free applications. The point being, is that the results are not for all android users, only the ones who have downloaded Geekbench, which is a small percentage.

Analyzing the GS3 results further

When Geekbench reports the score for a particular device, it uses the average for all users of that device. See here, for the individual android results for the Samsung Galaxy S3 phone.

Primelabs Results for Galaxy S3

The Samsung Galaxy S3 is an interesting one, because, depending on the country it is sold in, it varies from 2 to 4 cores. As you can see, from the link above, there is a big variance from 1250 to 2300. But thats not all, the variance also differs for the same user on the same phone. What is going on? Is Geekbench as bad benchmarking suite? Maybe, but that what Mac owners fail to realize, is that Androids are heavily customizable.

Android owners can, after rooting their devices, overclock them, install custom roms. Even without rooting, they can change the launcher (i.e. desktop), to not only get a different look and feel, but from the Geekbench results, a different benchmarking result. To prove this point, here is one of the comments for GeekBench 2 on Google Play

Joe – September 17, 2012 – Samsung Galaxy S3 with version 2.3.4
Get odd results with my s3. With go launcher ex running I get 1240 to 1300. With TouchWiz I get 1680. Movie mg back to go launcher I get 1890 or so. Strange variance I thought.

Also, there may actually be a problem with Geekbench, at least on some phones.

Sam – September 20, 2012
Inaccurate.
It’s short and easy to do but it has recorded my phone getting anywhere between like 600 – 1800… so obviously something is wrong… I get 8k give or take on Antutu constantly. Not sure what’s wrong but my results are pretty spastic. Dual core at 1.5ghz (or more)should not be the same or worse than a single core at 1ghz lol.

Ivn – September 17, 2012 – HTC One X with version 2.3.4
Weird
Everytime I get different results, I have other benchmarks like quadrand,gl,antutu all shows similar +- 50 results. This one is like sometimes 1280 and other time 1650

GS3s are faster on Geekbench

However, when I last looked at the results for Android, it was faster. iOS is 1589, and android is 1754 (SG3).

 

 

 

You can view the current results here

android-benchmarks

ios-benchmarks

So why has SG3 jumped up, without any major changes. Note that at the time of writing, the Jellybean update is not available officially. A clue may be in the number of installs reported on the GeekBench Google play page. You can that there is a jump recently from hardly any installs to a big jump.

One conclusion that can be drawn, is that the early benchmarks for SG3, may have come for users who customize their phones extensively. They maybe using roms that are still in development, or beta stages, or they are using launchers that give bad GeekBench results. The latest jump in figures, maybe that a lot of stock, unmodified android phones are being used, thus giving a more accurate and higher set of results.

But as to the future, there’s another 12 or so months before the next iPhone comes out. A year in the Android world is a long time. More Androids handsets will be released, with better processors, Arm A15, Qualcomm S4 Pro, etc. They will also be released with Jellybean, the latest Android OS, on which a large part of the development was spent on making Android faster. So we may see even higher results for Android in the future.