Weight quantization is an effective technique to compress deep neural
networks for their deployment on edge devices with limited resources.
Traditional loss-aware quantization methods commonly use the quantized gradient
to replace the full-precision gradient. However, we discover that the gradient
error will lead to an unexpected zig-zagging-like issue in the gradient descent
learning procedures, where the gradient directions rapidly oscillate or
zig-zag, and such issue seriously slows down the model convergence.
Accordingly, this paper proposes a one-step forward and backtrack way for
loss-aware quantization to get more accurate and stable gradient direction to
defy this issue. During the gradient descent learning, a one-step forward
search is designed to find the trial gradient of the next-step, which is
adopted to adjust the gradient of current step towards the direction of fast
convergence. After that, we backtrack the current step to update the
full-precision and quantized weights through the current-step gradient and the
trial gradient. A series of theoretical analysis and experiments on benchmark
deep models have demonstrated the effectiveness and competitiveness of the
proposed method, and our method especially outperforms others on the
convergence performance.

本文提出了一种新的损失感知量化方法，通过一步前进和回溯的方式来获取更准确、更稳定的梯度方向，解决了梯度下降学习过程中梯度方向快速震荡的问题，实验证明该方法在收敛性能上具有竞争力且优于其他方法。