Low-precision arithmetic trains deep learning models using less energy, less
memory and less time. However, we pay a price for the savings: lower precision
may yield larger round-off error and hence larger prediction error. As
applications proliferate, users must choose which precision to use to train a
new model, and chip manufacturers must decide which precisions to manufacture.
We view these precision choices as a hyperparameter tuning problem, and borrow
ideas from meta-learning to learn the tradeoff between memory and error. In
this paper, we introduce Pareto Estimation to Pick the Perfect Precision
(PEPPP). We use matrix factorization to find non-dominated configurations (the
Pareto frontier) with a limited number of network evaluations. For any given
memory budget, the precision that minimizes error is a point on this frontier.
Practitioners can use the frontier to trade memory for error and choose the
best precision for their goals.

本文提出一种基于 Pareto 估计的技术 (Pareto Estimation to Pick the Perfect Precision，PEPPP)，使用矩阵分解来查找非支配配置 (即 Pareto 前沿)，以在有限数量的网络评估下找到最小化错误的精度，从而解决低精度算术训练深度学习模型中的超参数调节问题。

低精度训练中的内存与误差权衡

How Low Can We Go: Trading Memory for Error in Low-Precision Training

Network quantization allows inference to be conducted using low-precision
arithmetic for improved inference efficiency of deep neural networks on edge
devices. However, designing aggressively low-bit (e.g., 2-bit) quantization
schemes on complex tasks, such as object detection, still remains challenging
in terms of severe performance degradation and unverifiable efficiency on
common hardware. In this paper, we propose an Accurate Quantized object
Detection solution, termed AQD, to fully get rid of floating-point computation.
To this end, we target using fixed-point operations in all kinds of layers,
including the convolutional layers, normalization layers, and skip connections,
allowing the inference to be executed using integer-only arithmetic. To
demonstrate the improved latency-vs-accuracy trade-off, we apply the proposed
methods on RetinaNet and FCOS. In particular, experimental results on MS-COCO
dataset show that our AQD achieves comparable or even better performance
compared with the full-precision counterpart under extremely low-bit schemes,
which is of great practical value. Source code and models are available at:
this https URL

本文提出了一种名为 AQD 的准确量化目标检测解决方案，采用包括卷积层、归一化层和跳跃连接在内的所有层的定点运算，在极低位方案下实现与全精度方案相媲美甚至更好的性能表现。