Laplace approximation (LA) and its linearized variant (LLA) enable effortless
adaptation of pretrained deep neural networks to Bayesian neural networks. The
generalized Gauss-Newton (GGN) approximation is typically introduced to improve
their tractability. However, LA and LLA are still confronted with non-trivial
inefficiency issues and should rely on Kronecker-factored, diagonal, or even
last-layer approximate GGN matrices in practical use. These approximations are
likely to harm the fidelity of learning outcomes. To tackle this issue,
inspired by the connections between LLA and neural tangent kernels (NTKs), we
develop a Nystrom approximation to NTKs to accelerate LLA. Our method benefits
from the capability of popular deep learning libraries for forward mode
automatic differentiation, and enjoys reassuring theoretical guarantees.
Extensive studies reflect the merits of the proposed method in aspects of both
scalability and performance. Our method can even scale up to architectures like
vision transformers. We also offer valuable ablation studies to diagnose our
method. Code is available at https://github.com/thudzj/ELLA.

通过开发一种 Nystrom 近似方法来加速线性化变种 (Laplace Approximation) 和神经切向核 (NTKs) 之间的联系，以解决 Bayesian 神经网络中的非常规低效率问题。该方法通过自动区分前向模式来实现，具有可靠的理论保证，并在规模性和性能方面表现出许多优点。

贝叶斯深度学习的加速线性化拉普拉斯近似

Accelerated Linearized Laplace Approximation for Bayesian Deep Learning

The generalized Gauss-Newton (GGN) approximation is often used to make
practical Bayesian deep learning approaches scalable by replacing a second
order derivative with a product of first order derivatives. In this paper we
argue that the GGN approximation should be understood as a local linearization
of the underlying Bayesian neural network (BNN), which turns the BNN into a
generalized linear model (GLM). Because we use this linearized model for
posterior inference, we should also predict using this modified model instead
of the original one. We refer to this modified predictive as "GLM predictive"
and show that it effectively resolves common underfitting problems of the
Laplace approximation. It extends previous results in this vein to general
likelihoods and has an equivalent Gaussian process formulation, which enables
alternative inference schemes for BNNs in function space. We demonstrate the
effectiveness of our approach on several standard classification datasets as
well as on out-of-distribution detection. We provide an implementation at
this https URL

本文提出了一种基于广义高斯牛顿近似方法的贝叶斯神经网络预测方法，将原始预测模型线性化为广义线性模型（GLM）后，用于后验推理和预测中，解决了拉普拉斯近似方法下的欠拟合问题。在多个标准分类数据集上以及外部分布检测中得到了验证。