Uncertainty quantification for deep neural networks has recently evolved through many techniques. In this work, we revisit Laplace approximation, a classical approach for posterior approximation that is computationally attractive. However, instead of computing the curvature matrix, we show that, under some regularity conditions, the Laplace approximation can be easily constructed using the gradient second moment. This quantity is already estimated by many exponential moving average variants of Adagrad such as Adam and RMSprop, but is traditionally discarded after training. We show that our method (L2M) does not require changes in models or optimization, can be implemented in a few lines of code to yield reasonable results, and it does not require any extra computational steps besides what is already being computed by optimizers, without introducing any new hyperparameter. We hope our method can open new research directions on using quantities already computed by optimizers for uncertainty estimation in deep neural networks.

本文提出了一种新的方法（L2M），通过使用Adagrad等优化器已经估算出来的梯度二次矩来构造Laplace近似，而不需要计算曲率矩阵。该方法不需要改变模型或优化器，可以通过几行代码实现，并且不需要引入任何新的超参数。我们希望该方法能为深度神经网络的不确定性估计开辟新的研究方向。

L2M：基于优化驱动的二阶矩估计的实用后验拉普拉斯近似方法