Low-precision training has emerged as a promising low-cost technique to
enhance the training efficiency of deep neural networks without sacrificing
much accuracy. Its Bayesian counterpart can further provide uncertainty
quantification and improved generalization accuracy. This paper investigates
low-precision sampling via Stochastic Gradient Hamiltonian Monte Carlo (SGHMC)
with low-precision and full-precision gradient accumulators for both strongly
log-concave and non-log-concave distributions. Theoretically, our results show
that, to achieve $\epsilon$-error in the 2-Wasserstein distance for
non-log-concave distributions, low-precision SGHMC achieves quadratic
improvement
($\widetilde{\mathbf{O}}\left({\epsilon^{-2}{\mu^*}^{-2}\log^2\left({\epsilon^{-1}}\right)}\right)$)
compared to the state-of-the-art low-precision sampler, Stochastic Gradient
Langevin Dynamics (SGLD)
($\widetilde{\mathbf{O}}\left({{\epsilon}^{-4}{\lambda^{*}}^{-1}\log^5\left({\epsilon^{-1}}\right)}\right)$).
Moreover, we prove that low-precision SGHMC is more robust to the quantization
error compared to low-precision SGLD due to the robustness of the
momentum-based update w.r.t. gradient noise. Empirically, we conduct
experiments on synthetic data, and {MNIST, CIFAR-10 \& CIFAR-100} datasets,
which validate our theoretical findings. Our study highlights the potential of
low-precision SGHMC as an efficient and accurate sampling method for
large-scale and resource-limited machine learning.

低精度训练是一种具有低成本的技术，可以提高深度神经网络的训练效率，而不会牺牲太多准确性。本文通过在强对数凹和非对数凹分布中使用低精度和全精度梯度累加器的随机梯度哈密顿蒙特卡罗（SGHMC）进行低精度采样的研究。理论上，我们的结果表明，对于非对数凹分布，在 2-Wasserstein 距离中实现 ε- 错误，与最先进的低精度采样器随机梯度朗之万动力学（SGLD）相比，低精度 SGHMC 实现了二次改进（〜O（ε^ -2μ^ -2 log^2（ε^ -1）））。此外，我们证明了相对于低精度的 SGLD，低精度的 SGHMC 对量化误差更具鲁棒性，因为基于动量的更新对梯度噪音具有鲁棒性。实验上，我们在合成数据，MNIST、CIFAR-10 和 CIFAR-100 数据集上进行了实验，验证了我们的理论发现。我们的研究突出了低精度 SGHMC 作为一种高效精确的大规模和资源有限的机器学习采样方法的潜力。

通过随机梯度哈密顿蒙特卡罗增强低精度采样

Enhancing Low-Precision Sampling via Stochastic Gradient Hamiltonian  Monte Carlo

Embedding tables are usually huge in click-through rate (CTR) prediction
models. To train and deploy the CTR models efficiently and economically, it is
necessary to compress their embedding tables at the training stage. To this
end, we formulate a novel quantization training paradigm to compress the
embeddings from the training stage, termed low-precision training (LPT). Also,
we provide theoretical analysis on its convergence. The results show that
stochastic weight quantization has a faster convergence rate and a smaller
convergence error than deterministic weight quantization in LPT. Further, to
reduce the accuracy degradation, we propose adaptive low-precision training
(ALPT) that learns the step size (i.e., the quantization resolution) through
gradient descent. Experiments on two real-world datasets confirm our analysis
and show that ALPT can significantly improve the prediction accuracy,
especially at extremely low bit widths. For the first time in CTR models, we
successfully train 8-bit embeddings without sacrificing prediction accuracy.
The code of ALPT is publicly available.

通过低精度训练和自适应的低精度训练方法，成功地压缩了 CTR 预测模型中的 embedding tables，并在不损失预测准确性的情况下实现了 8 位 embedding tables 的训练。

面向 CTR 预测的自适应低精度嵌入训练

Adaptive Low-Precision Training for Embeddings in Click-Through Rate  Prediction

Low-precision deep neural network (DNN) training has gained tremendous
attention as reducing precision is one of the most effective knobs for boosting
DNNs' training time/energy efficiency. In this paper, we attempt to explore
low-precision training from a new perspective as inspired by recent findings in
understanding DNN training: we conjecture that DNNs' precision might have a
similar effect as the learning rate during DNN training, and advocate dynamic
precision along the training trajectory for further boosting the time/energy
efficiency of DNN training. Specifically, we propose Cyclic Precision Training
(CPT) to cyclically vary the precision between two boundary values which can be
identified using a simple precision range test within the first few training
epochs. Extensive simulations and ablation studies on five datasets and eleven
models demonstrate that CPT's effectiveness is consistent across various
models/tasks (including classification and language modeling). Furthermore,
through experiments and visualization we show that CPT helps to (1) converge to
a wider minima with a lower generalization error and (2) reduce training
variance which we believe opens up a new design knob for simultaneously
improving the optimization and efficiency of DNN training. Our codes are
available at: this https URL

本文探索了一种名为 Cyclic Precision Training (CPT) 的新型低精度训练方法，其能够提升深度神经网络训练的时间和能量效率，并在五个数据集和十一种模型上进行了大量仿真和消融研究。

CPT: 基于循环精度的高效深度神经网络训练

CPT: Efficient Deep Neural Network Training via Cyclic Precision

Low-precision training reduces computational cost and produces efficient
models. Recent research in developing new low-precision training algorithms
often relies on simulation to empirically evaluate the statistical effects of
quantization while avoiding the substantial overhead of building specific
hardware. To support this empirical research, we introduce QPyTorch, a
low-precision arithmetic simulation framework. Built natively in PyTorch,
QPyTorch provides a convenient interface that minimizes the efforts needed to
reliably convert existing codes to study low-precision training. QPyTorch is
general, and supports a variety of combinations of precisions, number formats,
and rounding options. Additionally, it leverages an efficient fused-kernel
approach to reduce simulator overhead, which enables simulation of large-scale,
realistic problems. QPyTorch is publicly available at
this https URL

本文介绍了 QPyTorch 模拟框架，利用该框架可以方便地研究低精度训练。通过使用该框架，可以有效减小低精度训练给计算机带来的负担，以及避免硬件开发的高昂成本，从而得到效率更高的模型。