INSUBCONTINENT EXCLUSIVE:
A group of computer vision researchers from ETH Zurich want to do their bit to enhance AI development on smartphones
To wit: They&ve created a benchmark system for assessing the performance of several major neural network architectures used for common AI
tasks.
They&re hoping it will be useful to other AI researchers but also to chipmakers (by helping them get competitive insights); Android
developers (to see how fast their AI models will run on different devices); and, well, to phone nerds — such as by showing whether or not
a particular device contains the necessary drivers for AI accelerators
(And, therefore, whether or not they should believe a company marketing messages.)
The app, called AI Benchmark, is available for download
on Google Play and can run on any devicewith Android 4.1 or higher — generating a score the researchers describe as a &final verdict& of
the device AI performance.
AI tasks being assessed by their benchmark system include image classification, face recognition, image
deblurring, image super-resolution, photo enhancement or segmentation.
They are even testingsome algorithms used in autonomous driving
systems, though there not really any practical purpose for doing that at this point
(Looking down the road, the researchers say it not clear what hardware platform will be used for autonomous driving — and they suggest it
&quite possible& mobile processors will, in future, become fast enough to be used for this task
So they&re at least prepped for that possibility.)
The app also includes visualizations of the algorithms& output to help users assess the
results and get a feel for the current state-of-the-art in various AI fields.
The researchers hope their score will become a universally
accepted metric — similar to DxOMark that is used for evaluating camera performance — and all algorithms included in the benchmark are
The current ranking of different smartphones and mobile processors is available on the project webpage.
The benchmark system and app was
around three months in development, says AI researcher and developer Andrey Ignatov.
He explains that the score being displayed reflects two
main aspects: The SoC speed and available RAM.
Let consider two devices: one with a score of 6000 and one with a score of 200
If some AI algorithm will run on the first device for 5 seconds, then this means that on the second device this will take about 30 times
And if we are thinking about applications like face recognition this is not just about the speed, but about the applicability of the
approach: Nobody will wait 10 seconds till their phone will be trying to recognize them.
The same is about memory: The larger is the
network/input image — the more RAM is needed to process it
If the phone has small amount of RAM that is e.g
only enough to enhance 0.3MP photo, then this enhancement will be clearly useless, but if it can do the same job for Full HD images — this
opens up much wider possibilities
So, basically the higher score — the more complex algorithms can be used / larger images can be processed / it will take less time to do
this.
Discussing the idea for the benchmark, Ignatov says the lab is ''tightly bound& to both research and industry — soat some point we
became curious about what are the limitations of running the recent AI algorithms on smartphones&.
Since there was no information about this
(currently, all AI algorithms are running remotely on the servers, not on your device, except for some built-in apps integrated in phone
firmware), we decided to develop our own tool that will clearly show the performance and capabilities of each device,& he adds.
We can say
that we are quite satisfied with the obtained results — despite all current problems, the industry is clearly moving towards using AI on
smartphones, and we also hope that our efforts will help to accelerate this movement and give some useful information for other members
participating in this development.
After building the benchmarking system and collating scores on a bunch of Android devices, Ignatov sums
up the current situation of AI on smartphones as &both interesting and absurd&.
For example, the team found that devices running Qualcomm
chips weren''t the clear winners they&d imagined — i.e
based on the company promotional materials about Snapdragon 845 AI capabilities and 8x performance acceleration.
It turned out that this
acceleration is available only for ‘quantized& networks that currently cannot be deployed on the phones, thus for ‘normal& networks you
won''t get any acceleration at all,& he says
&The saddest thing is that actually they can theoretically provide acceleration for the latter networks too, but they just haven''t
implemented the appropriated drivers yet, and the only possible way to get this acceleration now is to use Snapdragon proprietary SDK
available for their own processors only
As a result — if you are developing an app that is using AI, you won''t get any acceleration on Snapdragon SoCs, unless you are developing
it for their processors only.
Whereas the researchers found that Huawei Kirin 970 CPU — which is technically even slower than Snapdragon
636 — offered a surprisingly strong performance.
Their integrated NPU gives almost 10x acceleration for Neural Networks, and thus even the
most powerful phone CPUs and GPUs can''t compete with it,& says Ignatov
&Additionally, Huawei P20/P20 Pro are the only smartphones on the market running Android 8.1 that are currently providing AI acceleration,
all other phones will get this support only in Android 9 or later.
It not all great news for Huawei phone owners, though, as Ignatov says
the NPUdoesn''t provide acceleration for ‘quantized& networks (though he notes the company has promised to add this support by the end of
this year); and also it uses its own RAM — which is &quite limited& in size, and therefore you &can''t process large images with it&…
We
would say that if they solve these two issues — most likely nobody will be able to compete with them within the following year(s),& he
suggests, though he also emphasizes that this assessmentonly refers to the one SoC, noting that Huawei processors don''t have the NPU
module.
For Samsung processors, the researchers flag up thatall the company devices are still running Android 8.0 but AI acceleration is
only available starting from Android 8.1 and above
Natch.
They also found CPU performance could &vary quite significantly& — up to 50% on the same Samsung device — because of throttling
and power optimization logic
Which would then have a knock on impact on AI performance.
ForMediatek, the researchers found the chipmaker isproviding acceleration for
both ‘quantized& and ‘normal& networks — which means it can reach the performance of ''top CPUs&.
But, on the flip side,Ignatov calls
out the company slogan — that it &Leading the Edge-AI Technology Revolution& — dubbing it ¬hing more than their dream&, and adding:
&Even the aforementioned Samsung latest Exynos CPU can slightly outperform it without using any acceleration at all, not to mention Huawei
with its Kirin 970 NPU.
In summary: Snapdragon — can theoretically provide good results, but are lacking the drivers; Huawei — quite
outstanding results now and most probably in the nearest future; Samsung — no acceleration support now (most likely this will change soon
since they are now developing their own AI Chip), but powerful CPUs; Mediatek — good results for mid-range devices, but definitely no
breakthrough.
It also worth noting that some of the results were obtained onprototype samples, rather than shipped smartphones, so haven''t
yet been included in the benchmark table on the team website.
We will wait till the devices with final firmware will come to the market
since some changes might still be introduced,& he adds.
For more on the pros and cons of AI-powered smartphone features check out our
article from earlier this year.
