Low-precision training has emerged as a promising low-cost technique to
enhance the training efficiency of deep neural networks without sacrificing
much accuracy. Its Bayesian counterpart can further provide uncertainty
quantification and improved generalization accuracy. This paper investigates
low-precision sampling via Stochastic Gradient Hamiltonian Monte Carlo (SGHMC)
with low-precision and full-precision gradient accumulators for both strongly
log-concave and non-log-concave distributions. Theoretically, our results show
that, to achieve $\epsilon$-error in the 2-Wasserstein distance for
non-log-concave distributions, low-precision SGHMC achieves quadratic
improvement
($\widetilde{\mathbf{O}}\left({\epsilon^{-2}{\mu^*}^{-2}\log^2\left({\epsilon^{-1}}\right)}\right)$)
compared to the state-of-the-art low-precision sampler, Stochastic Gradient
Langevin Dynamics (SGLD)
($\widetilde{\mathbf{O}}\left({{\epsilon}^{-4}{\lambda^{*}}^{-1}\log^5\left({\epsilon^{-1}}\right)}\right)$).
Moreover, we prove that low-precision SGHMC is more robust to the quantization
error compared to low-precision SGLD due to the robustness of the
momentum-based update w.r.t. gradient noise. Empirically, we conduct
experiments on synthetic data, and {MNIST, CIFAR-10 \& CIFAR-100} datasets,
which validate our theoretical findings. Our study highlights the potential of
low-precision SGHMC as an efficient and accurate sampling method for
large-scale and resource-limited machine learning.

低精度训练是一种具有低成本的技术，可以提高深度神经网络的训练效率，而不会牺牲太多准确性。本文通过在强对数凹和非对数凹分布中使用低精度和全精度梯度累加器的随机梯度哈密顿蒙特卡罗（SGHMC）进行低精度采样的研究。理论上，我们的结果表明，对于非对数凹分布，在 2-Wasserstein 距离中实现 ε- 错误，与最先进的低精度采样器随机梯度朗之万动力学（SGLD）相比，低精度 SGHMC 实现了二次改进（〜O（ε^ -2μ^ -2 log^2（ε^ -1）））。此外，我们证明了相对于低精度的 SGLD，低精度的 SGHMC 对量化误差更具鲁棒性，因为基于动量的更新对梯度噪音具有鲁棒性。实验上，我们在合成数据，MNIST、CIFAR-10 和 CIFAR-100 数据集上进行了实验，验证了我们的理论发现。我们的研究突出了低精度 SGHMC 作为一种高效精确的大规模和资源有限的机器学习采样方法的潜力。