Deep neural networks with adaptive configurations have gained increasing
attention due to the instant and flexible deployment of these models on
platforms with different resource budgets. In this paper, we investigate a
novel option to achieve this goal by enabling adaptive bit-widths of weights
and activations in the model. We first examine the benefits and challenges of
training quantized model with adaptive bit-widths, and then experiment with
several approaches including direct adaptation, progressive training and joint
training. We discover that joint training is able to produce comparable
performance on the adaptive model as individual models. We further propose a
new technique named Switchable Clipping Level (S-CL) to further improve
quantized models at the lowest bit-width. With our proposed techniques applied
on a bunch of models including MobileNet-V1/V2 and ResNet-50, we demonstrate
that bit-width of weights and activations is a new option for adaptively
executable deep neural networks, offering a distinct opportunity for improved
accuracy-efficiency trade-off as well as instant adaptation according to the
platform constraints in real-world applications.

本文研究了一种通过使权重和激活的位宽自适应来实现神经网络自适应配置的新方法，并探究了直接适应、渐进适应和联合训练等多种方法，最终提出了一种称为 S-CL 的新技术，证明了通过位宽自适应可以在实现高精度的同时实现高效率，从而在真实世界应用中快速适配。