On-device inference of machine learning models for mobile phones is desirable
due to its lower latency and increased privacy. Running such a
compute-intensive task solely on the mobile CPU, however, can be difficult due
to limited computing power, thermal constraints, and energy consumption. App
developers and researchers have begun exploiting hardware accelerators to
overcome these challenges. Recently, device manufacturers are adding neural
processing units into high-end phones for on-device inference, but these
account for only a small fraction of hand-held devices. In this paper, we
present how we leverage the mobile GPU, a ubiquitous hardware accelerator on
virtually every phone, to run inference of deep neural networks in real-time
for both Android and iOS devices. By describing our architecture, we also
discuss how to design networks that are mobile GPU-friendly. Our
state-of-the-art mobile GPU inference engine is integrated into the open-source
project TensorFlow Lite and publicly available at this https URL

本文介绍了如何利用手机上普遍存在的 GPU 加速器，在 Android 和 iOS 设备上实现深度神经网络的实时推断，并将其集成到开源项目 TensorFlow Lite 中。