Quantization-aware training (QAT) simulates a quantization process during
training to lower bit-precision of weights/activations. It learns quantized
weights indirectly by updating latent weights, i.e., full-precision inputs to a
quantizer, using gradient-based optimizers. We claim that coupling a
user-defined learning rate (LR) with these optimizers is sub-optimal for QAT.
Quantized weights transit discrete levels of a quantizer, only if corresponding
latent weights pass transition points, where the quantizer changes discrete
states. This suggests that the changes of quantized weights are affected by
both the LR for latent weights and their distributions. It is thus difficult to
control the degree of changes for quantized weights by scheduling the LR
manually. We conjecture that the degree of parameter changes in QAT is related
to the number of quantized weights transiting discrete levels. Based on this,
we introduce a transition rate (TR) scheduling technique that controls the
number of transitions of quantized weights explicitly. Instead of scheduling a
LR for latent weights, we schedule a target TR of quantized weights, and update
the latent weights with a novel transition-adaptive LR (TALR), enabling
considering the degree of changes for the quantized weights during QAT.
Experimental results demonstrate the effectiveness of our approach on standard
benchmarks.

基于量化意识训练 (QAT) 的过程中，通过控制权重 / 激活的位精度来学习量化权重，通过更新潜在权重间接学习，本文提出了一种基于过渡速率 (transition rate) 的调度技术，通过调整量化权重的过渡点数目控制量化权重的变化程度，取得了显著的效果。

量化感知训练的转换率调度

Transition Rate Scheduling for Quantization-Aware Training

Quantization methods reduce the number of bits required to represent each
parameter in a model, trading accuracy for smaller memory footprints and
inference latencies. However, the final model size depends on both the number
of parameters of the original model and the rate of compression. For example, a
30B 8-bit model and a 60B 4-bit model have the same number of bits but may have
very different zero-shot accuracies. In this work, we study this trade-off by
developing inference scaling laws of zero-shot performance in Large Language
Models (LLMs) to determine the bit-precision and model size that maximizes
zero-shot performance. We run more than 35,000 experiments with 16-bit inputs
and k-bit parameters to examine which zero-shot quantization methods improve
scaling for 3 to 8-bit precision at scales of 19M to 176B parameters across the
LLM families BLOOM, OPT, NeoX/Pythia, and GPT-2. We find that it is challenging
to improve the bit-level scaling trade-off, with the only improvements being
the use of a small block size -- splitting the parameters into small
independently quantized blocks -- and the quantization data type being used
(e.g., Int vs Float). Overall, our findings show that {4-bit} precision is
almost universally optimal for total model bits and zero-shot accuracy.

研究语言模型在压缩模型时的精度与模型大小之间的平衡，发现在总模型位和零 - shot 准确度方面，{4 位} 精度几乎普遍最优。

4 位精度的必要性：k 位推断缩放定律

The case for 4-bit precision: k-bit Inference Scaling Laws

Deep neural networks are gaining in popularity as they are used to generate
state-of-the-art results for a variety of computer vision and machine learning
applications. At the same time, these networks have grown in depth and
complexity in order to solve harder problems. Given the limitations in power
budgets dedicated to these networks, the importance of low-power, low-memory
solutions has been stressed in recent years. While a large number of dedicated
hardware using different precisions has recently been proposed, there exists no
comprehensive study of different bit precisions and arithmetic in both inputs
and network parameters. In this work, we address this issue and perform a study
of different bit-precisions in neural networks (from floating-point to
fixed-point, powers of two, and binary). In our evaluation, we consider and
analyze the effect of precision scaling on both network accuracy and hardware
metrics including memory footprint, power and energy consumption, and design
area. We also investigate training-time methodologies to compensate for the
reduction in accuracy due to limited bit precision and demonstrate that in most
cases, precision scaling can deliver significant benefits in design metrics at
the cost of very modest decreases in network accuracy. In addition, we propose
that a small portion of the benefits achieved when using lower precisions can
be forfeited to increase the network size and therefore the accuracy. We
evaluate our experiments, using three well-recognized networks and datasets to
show its generality. We investigate the trade-offs and highlight the benefits
of using lower precisions in terms of energy and memory footprint.

本研究探讨神经网络中不同位精度 (bit-precision) 对网络精度、硬件指标 (存储、功耗、设计面积)、能量消耗等的影响，并提出训练时间方法来弥补位精度降低精度所导致的误差，并表明在降低精度的情况下，可以在设计指标中获得显著的益处，同时提出可以通过增加网络大小来提高精度。