The rise of mobile AI accelerators allows latency-sensitive applications to
execute lightweight Deep Neural Networks (DNNs) on the client side. However,
critical applications require powerful models that edge devices cannot host and
must therefore offload requests, where the high-dimensional data will compete
for limited bandwidth. This work proposes shifting away from focusing on
executing shallow layers of partitioned DNNs. Instead, it advocates
concentrating the local resources on variational compression optimized for
machine interpretability. We introduce a novel framework for resource-conscious
compression models and extensively evaluate our method in an environment
reflecting the asymmetric resource distribution between edge devices and
servers. Our method achieves 60\% lower bitrate than a state-of-the-art SC
method without decreasing accuracy and is up to 16x faster than offloading with
existing codec standards.

本文提出了一种面向资源的压缩模型框架，旨在提高机器解释性能，解决边缘设备和服务器之间资源分配不对称的问题，实验结果表明，相比现有的编解码标准任务，本方法可以获得 60％的比特率降低，同时不降低准确性，并可以比离线处理快 16 倍。

FrankenSplit: 基于显著性引导的神经特征压缩与浅层变分瓶颈注入

FrankenSplit: Saliency Guided Neural Feature Compression with Shallow Variational Bottleneck Injection

The performance of mobile AI accelerators has been evolving rapidly in the
past two years, nearly doubling with each new generation of SoCs. The current
4th generation of mobile NPUs is already approaching the results of
CUDA-compatible Nvidia graphics cards presented not long ago, which together
with the increased capabilities of mobile deep learning frameworks makes it
possible to run complex and deep AI models on mobile devices. In this paper, we
evaluate the performance and compare the results of all chipsets from Qualcomm,
HiSilicon, Samsung, MediaTek and Unisoc that are providing hardware
acceleration for AI inference. We also discuss the recent changes in the
Android ML pipeline and provide an overview of the deployment of deep learning
models on mobile devices. All numerical results provided in this paper can be
found and are regularly updated on the official project website:
this http URL

本文评估了高通、海思、三星、联发科和紫光展锐提供的 AI 推理硬件加速的所有芯片组的性能，并比较了它们的结果。同时，我们还讨论了 Android ML 流水线的最近变化，并提供了在移动设备上部署深度学习模型的概述。