We tackle the problem of producing compact models, maximizing their accuracy
for a given model size. A standard solution is to train networks with
Quantization Aware Training, where the weights are quantized during training
and the gradients approximated with the Straight-Through Estimator. In this
paper, we extend this approach to work beyond int8 fixed-point quantization
with extreme compression methods where the approximations introduced by STE are
severe, such as Product Quantization. Our proposal is to only quantize a
different random subset of weights during each forward, allowing for unbiased
gradients to flow through the other weights. Controlling the amount of noise
and its form allows for extreme compression rates while maintaining the
performance of the original model. As a result we establish new
state-of-the-art compromises between accuracy and model size both in natural
language processing and image classification. For example, applying our method
to state-of-the-art Transformer and ConvNet architectures, we can achieve 82.5%
accuracy on MNLI by compressing RoBERTa to 14MB and 80.0 top-1 accuracy on
ImageNet by compressing an EfficientNet-B3 to 3.3MB.

本文针对在给定模型大小时最大化其准确性的紧凑模型生成问题，将讨论延伸量化感知训练的方法，实现了仅在每个前向传递过程中量化不同的一组随机权重，从而利用 SE 残差正向时间传递的无偏梯度实现极端压缩的目的，并在自然语言处理和图像分类领域分别取得了新的准确性与模型大小之间的最优折中表现。