Deploying deep neural networks on low-resource edge devices is challenging
due to their ever-increasing resource requirements. Recent investigations
propose multiplication-free neural networks to reduce computation and memory
consumption. Shift neural network is one of the most effective tools towards
these reductions. However, existing low-bit shift networks are not as accurate
as their full precision counterparts and cannot efficiently transfer to a wide
range of tasks due to their inherent design flaws. We propose DenseShift
network that exploits the following novel designs. First, we demonstrate that
the zero-weight values in low-bit shift networks are neither useful to the
model capacity nor simplify the model inference. Therefore, we propose to use a
zero-free shifting mechanism to simplify inference while increasing the model
capacity. Second, we design a new metric to measure the weight freezing issue
in training low-bit shift networks, and propose a sign-scale decomposition to
improve the training efficiency. Third, we propose the low-variance random
initialization strategy to improve the model's performance in transfer learning
scenarios. We run extensive experiments on various computer vision and speech
tasks. The experimental results show that DenseShift network significantly
outperforms existing low-bit multiplication-free networks and can achieve
competitive performance to the full-precision counterpart. It also exhibits
strong transfer learning performance with no drop in accuracy.

提出了一种名为 DenseShift 的神经网络，在低比特训练和转移学习方面具有优越的性能，通过零 - free 位移机制，新的度量标准和随机初始化策略，有效地简化了推理并提高了模型容量。

DenseShift：面向准确和可迁移的低比特移位网络

DenseShift: Towards Accurate and Transferable Low-Bit Shift Network

Recently low-bit (e.g., 8-bit) network quantization has been extensively
studied to accelerate the inference. Besides inference, low-bit training with
quantized gradients can further bring more considerable acceleration, since the
backward process is often computation-intensive. Unfortunately, the
inappropriate quantization of backward propagation usually makes the training
unstable and even crash. There lacks a successful unified low-bit training
framework that can support diverse networks on various tasks. In this paper, we
give an attempt to build a unified 8-bit (INT8) training framework for common
convolutional neural networks from the aspects of both accuracy and speed.
First, we empirically find the four distinctive characteristics of gradients,
which provide us insightful clues for gradient quantization. Then, we
theoretically give an in-depth analysis of the convergence bound and derive two
principles for stable INT8 training. Finally, we propose two universal
techniques, including Direction Sensitive Gradient Clipping that reduces the
direction deviation of gradients and Deviation Counteractive Learning Rate
Scaling that avoids illegal gradient update along the wrong direction. The
experiments show that our unified solution promises accurate and efficient INT8
training for a variety of networks and tasks, including MobileNetV2,
InceptionV3 and object detection that prior studies have never succeeded.
Moreover, it enjoys a strong flexibility to run on off-the-shelf hardware, and
reduces the training time by 22% on Pascal GPU without too much optimization
effort. We believe that this pioneering study will help lead the community
towards a fully unified INT8 training for convolutional neural networks.

本文提出了一个统一的 INT8 训练框架，基于四个梯度特征和两个稳定性原则，以及两种普适性技术，包括方向敏感梯度裁剪和偏移抵消学习率缩放，可以支持各种网络和任务，并在不使用太多优化方法的情况下将训练时间降低了 22％。