As AI applications for mobile devices become more prevalent, there is an
increasing need for faster execution and lower energy consumption for deep
learning model inference. Historically, the models run on mobile devices have
been smaller and simpler in comparison to large state-of-the-art research
models, which can only run on the cloud. However, cloud-only inference has
drawbacks such as increased network bandwidth consumption and higher latency.
In addition, cloud-only inference requires the input data (images, audio) to be
fully transferred to the cloud, creating concerns about potential privacy
breaches.
There is an alternative approach: shared mobile-cloud inference. Partial
inference is performed on the mobile in order to reduce the dimensionality of
the input data and arrive at a compact feature tensor, which is a latent space
representation of the input signal. The feature tensor is then transmitted to
the server for further inference. This strategy can reduce inference latency,
energy consumption, and network bandwidth usage, as well as provide privacy
protection, because the original signal never leaves the mobile. Further
performance gain can be achieved by compressing the feature tensor before its
transmission.

在移动设备上，利用深度学习模型进行 AI 应用程序时，采用共享的移动 - 云推理方法可以降低推理的延迟、能源消耗和网络带宽使用，同时提供隐私保护措施。