Weight oscillation is an undesirable side effect of quantization-aware
training, in which quantized weights frequently jump between two quantized
levels, resulting in training instability and a sub-optimal final model. We
discover that the learnable scaling factor, a widely-used $\textit{de facto}$
setting in quantization aggravates weight oscillation. In this study, we
investigate the connection between the learnable scaling factor and quantized
weight oscillation and use ViT as a case driver to illustrate the findings and
remedies. In addition, we also found that the interdependence between quantized
weights in $\textit{query}$ and $\textit{key}$ of a self-attention layer makes
ViT vulnerable to oscillation. We, therefore, propose three techniques
accordingly: statistical weight quantization ($\rm StatsQ$) to improve
quantization robustness compared to the prevalent learnable-scale-based method;
confidence-guided annealing ($\rm CGA$) that freezes the weights with
$\textit{high confidence}$ and calms the oscillating weights; and
$\textit{query}$-$\textit{key}$ reparameterization ($\rm QKR$) to resolve the
query-key intertwined oscillation and mitigate the resulting gradient
misestimation. Extensive experiments demonstrate that these proposed techniques
successfully abate weight oscillation and consistently achieve substantial
accuracy improvement on ImageNet. Specifically, our 2-bit DeiT-T/DeiT-S
algorithms outperform the previous state-of-the-art by 9.8% and 7.7%,
respectively. The code is included in the supplementary material and will be
released.

研究了量化感知训练中的重量振荡，发现可学习缩放因子会加剧重量振荡，并因此提出三种技术有效地减轻重量振荡并在 Imagenet 上显著提高了准确率。

低位视觉 Transformer 的无振荡量化

Oscillation-free Quantization for Low-bit Vision Transformers

Binary neural networks (BNNs) have received ever-increasing popularity for
their great capability of reducing storage burden as well as quickening
inference time. However, there is a severe performance drop compared with
real-valued networks, due to its intrinsic frequent weight oscillation during
training. In this paper, we introduce a Resilient Binary Neural Network (ReBNN)
to mitigate the frequent oscillation for better BNNs' training. We identify
that the weight oscillation mainly stems from the non-parametric scaling
factor. To address this issue, we propose to parameterize the scaling factor
and introduce a weighted reconstruction loss to build an adaptive training
objective. For the first time, we show that the weight oscillation is
controlled by the balanced parameter attached to the reconstruction loss, which
provides a theoretical foundation to parameterize it in back propagation. Based
on this, we learn our ReBNN by calculating the balanced parameter based on its
maximum magnitude, which can effectively mitigate the weight oscillation with a
resilient training process. Extensive experiments are conducted upon various
network models, such as ResNet and Faster-RCNN for computer vision, as well as
BERT for natural language processing. The results demonstrate the overwhelming
performance of our ReBNN over prior arts. For example, our ReBNN achieves 66.9%
Top-1 accuracy with ResNet-18 backbone on the ImageNet dataset, surpassing
existing state-of-the-arts by a significant margin. Our code is open-sourced at
this https URL

本文提出了一种 Resilient Binary Neural Network（ReBNN）来减轻二进制神经网络的重量震荡问题，并通过引入加权重建损失来构建自适应训练目标，该方法在计算机视觉等多种网络模型上具有优异的表现。