Uncertainty computation in deep learning is essential to design robust and reliable systems. Variational inference (VI) is a promising approach for such computation, but requires more effort to implement and execute compared to maximum-likelihood methods. In this paper, we propose new natural-gradient algorithms to reduce such efforts for Gaussian mean-field VI. Our algorithms can be implemented within the Adam optimizer by perturbing the network weights during gradient evaluations, and uncertainty estimates can be cheaply obtained by using the vector that adapts the learning rate. This requires lower memory, computation, and implementation effort than existing VI methods, while obtaining uncertainty estimates of comparable quality. Our empirical results confirm this and further suggest that the weight-perturbation in our algorithm could be useful for exploration in reinforcement learning and stochastic optimization.

本文提出了新的自然梯度算法来降低高斯平均场变分推断的实现难度，借助Adam优化器在梯度计算期间扰动网络权重，通过使用适应学习率的向量可以快速获得不确定性估计值。实验结果表明，我们的方法可以获得与现有变分推断相当的不确定性估计，并表明我们算法中的权重扰动可以用于强化学习和随机优化中的探索。

Adam中的权重扰动快速可扩展的贝叶斯深度学习