Combining the flexibility of deep learning with Bayesian uncertainty estimation has long been a goal in our field, and many modern approaches are based on variational Bayes. Unfortunately, one is forced to choose between overly simplistic variational families (e.g. fully factorized) or expensive and complicated inference procedures. We show that natural gradient ascent with adaptive weight noise can be interpreted as fitting a variational posterior to maximize the evidence lower bound (ELBO). This insight allows us to train full covariance, fully factorized, and matrix variate Gaussian variational posteriors using noisy versions of natural gradient, Adam, and K-FAC, respectively. On standard regression benchmarks, our noisy K-FAC algorithm makes better predictions and matches HMC's predictive variances better than existing methods. Its improved uncertainty estimates lead to more efficient exploration in the settings of active learning and intrinsic motivation for reinforcement learning.

本文研究了变分贝叶斯神经网络、自然梯度上升和卷积神经网络等相关技术，提出了基于噪声版本的自然梯度、Adam和K-FAC算法来训练全协方差、完全因式分解或矩阵变量高斯变分后验，并应用于回归测试中，发现该算法可以更好地预测和匹配存在方法的预测方差，进而用于主动学习和增强学习中实现更高效的探索。

噪声自然梯度作为变分推理