It is necessary to develop efficient DNNs deployed on edge devices with
limited computation resources. However, the compressed networks often execute
new tasks in the target domain, which is different from the source domain where
the original network is trained. It is important to investigate the robustness
of compressed networks in two types of data distribution shifts: domain shifts
and adversarial perturbations. In this study, we discover that compressed
models are less robust to distribution shifts than their original networks.
Interestingly, larger networks are more vulnerable to losing robustness than
smaller ones, even when they are compressed to a similar size as the smaller
networks. Furthermore, compact networks obtained by knowledge distillation are
much more robust to distribution shifts than pruned networks. Finally,
post-training quantization is a reliable method for achieving significant
robustness to distribution shifts, and it outperforms both pruned and distilled
models in terms of robustness.

压缩网络在面对域转移和对抗扰动时的稳健性较原始网络较低，较大的网络比较小的网络更容易丧失稳健性，知识蒸馏得到的紧凑网络在面对域转移时比剪枝网络更稳健，后训练量化是一种可靠的方法，能显著提高稳健性，胜过剪枝和蒸馏模型的稳健性。

面向边缘设备的压缩网络对分布转移的鲁棒性

Robustness to distribution shifts of compressed networks for edge  devices

We investigate the compression of deep neural networks by quantizing their
weights and activations into multiple binary bases, known as multi-bit networks
(MBNs), which accelerate the inference and reduce the storage for the
deployment on low-resource mobile and embedded platforms. We propose Adaptive
Loss-aware Quantization (ALQ), a new MBN quantization pipeline that is able to
achieve an average bitwidth below one-bit without notable loss in inference
accuracy. Unlike previous MBN quantization solutions that train a quantizer by
minimizing the error to reconstruct full precision weights, ALQ directly
minimizes the quantization-induced error on the loss function involving neither
gradient approximation nor full precision maintenance. ALQ also exploits
strategies including adaptive bitwidth, smooth bitwidth reduction, and
iterative trained quantization to allow a smaller network size without loss in
accuracy. Experiment results on popular image datasets show that ALQ
outperforms state-of-the-art compressed networks in terms of both storage and
accuracy. Code is available at this https URL

研究深度神经网络压缩，提出一种自适应损失感知量化方法 (ALQ) 来加速推理并在低资源移动和嵌入式平台上降低存储需求。实验结果表明，该方法在精度和存储方面优于最先进的压缩网络。