While low-precision optimization has been widely used to accelerate deep
learning, low-precision sampling remains largely unexplored. As a consequence,
sampling is simply infeasible in many large-scale scenarios, despite providing
remarkable benefits to generalization and uncertainty estimation for neural
networks. In this paper, we provide the first study of low-precision Stochastic
Gradient Langevin Dynamics (SGLD), showing that its costs can be significantly
reduced without sacrificing performance, due to its intrinsic ability to handle
system noise. We prove that the convergence of low-precision SGLD with
full-precision gradient accumulators is less affected by the quantization error
than its SGD counterpart in the strongly convex setting. To further enable
low-precision gradient accumulators, we develop a new quantization function for
SGLD that preserves the variance in each update step. We demonstrate that
low-precision SGLD achieves comparable performance to full-precision SGLD with
only 8 bits on a variety of deep learning tasks.

本文研究了低精度随机梯度 Langevin 动力学（SGLD）和其量化方法在深度学习中的应用，发现低精度梯度累积器通过新开发的量化函数可以实现较少的成本和可比的性能。

低精度随机梯度 Langevin 动力学

Low-Precision Stochastic Gradient Langevin Dynamics

Although deep convolutional neural network has been proved to efficiently
eliminate coding artifacts caused by the coarse quantization of traditional
codec, it's difficult to train any neural network in front of the encoder for
gradient's back-propagation. In this paper, we propose an end-to-end image
compression framework based on convolutional neural network to resolve the
problem of non-differentiability of the quantization function in the standard
codec. First, the feature description neural network is used to get a valid
description in the low-dimension space with respect to the ground-truth image
so that the amount of image data is greatly reduced for storage or
transmission. After image's valid description, standard image codec such as
JPEG is leveraged to further compress image, which leads to image's great
distortion and compression artifacts, especially blocking artifacts, detail
missing, blurring, and ringing artifacts. Then, we use a post-processing neural
network to remove these artifacts. Due to the challenge of directly learning a
non-linear function for a standard codec based on convolutional neural network,
we propose to learn a virtual codec neural network to approximate the
projection from the valid description image to the post-processed compressed
image, so that the gradient could be efficiently back-propagated from the
post-processing neural network to the feature description neural network during
training. Meanwhile, an advanced learning algorithm is proposed to train our
deep neural networks for compression. Obviously, the priority of the proposed
method is compatible with standard existing codecs and our learning strategy
can be easily extended into these codecs based on convolutional neural network.
Experimental results have demonstrated the advances of the proposed method as
compared to several state-of-the-art approaches, especially at very low
bit-rate.

本研究提出一种基于卷积神经网络的端到端图像压缩框架，通过特征描述神经网络获取低维有效描述，借助标准图像编解码器进一步压缩图像后，再通过后处理神经网络消除失真和压缩留痕，最终通过虚拟编解码器神经网络通过反向传播优化整个系统。实验结果表明该方法在极低比特率下比当前现有算法性能更优。