In today's landscape, smartphones have evolved into hubs for hosting a
multitude of deep learning models aimed at local execution. A key realization
driving this work is the notable fragmentation among these models,
characterized by varied architectures, operators, and implementations. This
fragmentation imposes a significant burden on the comprehensive optimization of
hardware, system settings, and algorithms.
Buoyed by the recent strides in large foundation models, this work introduces
a pioneering paradigm for mobile AI: a collaborative management approach
between the mobile OS and hardware, overseeing a foundational model capable of
serving a broad spectrum of mobile AI tasks, if not all. This foundational
model resides within the NPU and remains impervious to app or OS revisions,
akin to firmware. Concurrently, each app contributes a concise, offline
fine-tuned "adapter" tailored to distinct downstream tasks. From this concept
emerges a concrete instantiation known as \sys. It amalgamates a curated
selection of publicly available Large Language Models (LLMs) and facilitates
dynamic data flow. This concept's viability is substantiated through the
creation of an exhaustive benchmark encompassing 38 mobile AI tasks spanning 50
datasets, including domains such as Computer Vision (CV), Natural Language
Processing (NLP), audio, sensing, and multimodal inputs. Spanning this
benchmark, \sys unveils its impressive performance. It attains accuracy parity
in 85\% of tasks, demonstrates improved scalability in terms of storage and
memory, and offers satisfactory inference speed on Commercial Off-The-Shelf
(COTS) mobile devices fortified with NPU support. This stands in stark contrast
to task-specific models tailored for individual applications.

智能手机已成为承载多种深度学习模型的中心，该研究介绍了一种新的移动人工智能范式，使用协同管理方法在移动操作系统和硬件之间实现一个基础模型，能够为各种移动人工智能任务提供服务。

在 LLM 时代重新思考移动 AI 生态系统

Rethinking Mobile AI Ecosystem in the LLM Era

As the "Mobile AI" revolution continues to grow, so does the need to
understand the behaviour of edge-deployed deep neural networks. In particular,
MobileNets are the go-to family of deep convolutional neural networks (CNN) for
mobile. However, they often have significant accuracy degradation under
post-training quantization. While studies have introduced quantization-aware
training and other methods to tackle this challenge, there is limited
understanding into why MobileNets (and potentially depthwise-separable CNNs
(DWSCNN) in general) quantize so poorly compared to other CNN architectures.
Motivated to gain deeper insights into this phenomenon, we take a different
strategy and study the multi-scale distributional dynamics of MobileNet-V1, a
set of smaller DWSCNNs, and regular CNNs. Specifically, we investigate the
impact of quantization on the weight and activation distributional dynamics as
information propagates from layer to layer, as well as overall changes in
distributional dynamics at the network level. This fine-grained analysis
revealed significant dynamic range fluctuations and a "distributional mismatch"
between channelwise and layerwise distributions in DWSCNNs that lead to
increasing quantized degradation and distributional shift during information
propagation. Furthermore, analysis of the activation quantization errors show
that there is greater quantization error accumulation in DWSCNN compared to
regular CNNs. The hope is that such insights can lead to innovative strategies
for reducing such distributional dynamics changes and improve post-training
quantization for mobile.

通过分析 MobileNets 在量化后的分布动态中的不匹配性、动态范围波动、以及 DWSCNN 较传统 CNN 更易受到量化累积误差等现象，揭示了 DWSCNN 量化表现不佳的根本原因。这些认知或许能帮助未来发展出纠正此类问题，提高移动上的后训练量化表现的策略。

所有 MobileNets 都量化得不好吗？通过多尺度分布动力学视角深入探究量化对深度可分离卷积网络的影响

Do All MobileNets Quantize Poorly? Gaining Insights into the Effect of  Quantization on Depthwise Separable Convolutional Networks Through the Eyes  of Multi-scale Distributional Dynamics

Assuming hardware is the major constraint for enabling real-time mobile
intelligence, the industry has mainly dedicated their efforts to developing
specialized hardware accelerators for machine learning and inference. This
article challenges the assumption. By drawing on a recent real-time AI
optimization framework CoCoPIE, it maintains that with effective
compression-compiler co-design, it is possible to enable real-time artificial
intelligence on mainstream end devices without special hardware. CoCoPIE is a
software framework that holds numerous records on mobile AI: the first
framework that supports all main kinds of DNNs, from CNNs to RNNs, transformer,
language models, and so on; the fastest DNN pruning and acceleration framework,
up to 180X faster compared with current DNN pruning on other frameworks such as
TensorFlow-Lite; making many representative AI applications able to run in
real-time on off-the-shelf mobile devices that have been previously regarded
possible only with special hardware support; making off-the-shelf mobile
devices outperform a number of representative ASIC and FPGA solutions in terms
of energy efficiency and/or performance.

通过压缩 - 编译器共同设计可以在主流的移动设备上实现实时人工智能，而无需专用硬件支持。CoCoPIE 是一个支持所有主要 DNN 类型的框架，包括 CNN，RNN，转换器，语言模型等等，并且能够在现成的移动设备上运行许多典型的 AI 应用程序。在能源效率和性能方面，Off-the-shelf 移动设备也能胜过一些典型的 ASIC 和 FPGA 硬件解决方案。