Bayesian Neural Networks (BNNs) extend traditional neural networks to provide
uncertainties associated with their outputs. On the forward pass through a BNN,
predictions (and their uncertainties) are made either by Monte Carlo sampling
network weights from the learned posterior or by analytically propagating
statistical moments through the network. Though flexible, Monte Carlo sampling
is computationally expensive and can be infeasible or impractical under
resource constraints or for large networks. While moment propagation can
ameliorate the computational costs of BNN inference, it can be difficult or
impossible for networks with arbitrary nonlinearities, thereby restricting the
possible set of network layers permitted with such a scheme. In this work, we
demonstrate a simple yet effective approach for propagating statistical moments
through arbitrary nonlinearities with only 3 deterministic samples, enabling
few-sample variational inference of BNNs without restricting the set of network
layers used. Furthermore, we leverage this approach to demonstrate a novel
nonlinear activation function that we use to inject physics-informed prior
information into output nodes of a BNN.

本研究通过仅使用 3 个确定性样本来传播统计矩，实现了对任意非线性网络层的统计矩传播，从而使得少样本变分推断成为可能，并将此方法应用于一种新的非线性激活函数，用于向 Bayesian 神经网络的输出节点注入物理先验信息。

任意非线性贝叶斯神经网络的小样本变分推断

Few-sample Variational Inference of Bayesian Neural Networks with  Arbitrary Nonlinearities

Uncertainty quantification of neural networks is critical to measuring the
reliability and robustness of deep learning systems. However, this often
involves costly or inaccurate sampling methods and approximations. This paper
presents a sample-free moment propagation technique that propagates mean
vectors and covariance matrices across a network to accurately characterize the
input-output distributions of neural networks. A key enabler of our technique
is an analytic solution for the covariance of random variables passed through
nonlinear activation functions, such as Heaviside, ReLU, and GELU. The wide
applicability and merits of the proposed technique are shown in experiments
analyzing the input-output distributions of trained neural networks and
training Bayesian neural networks.

神经网络的不确定性量化对于评估深度学习系统的可靠性和鲁棒性至关重要。本文提出了一种无样本的矩阵传播技术，通过在网络中传播均值向量和协方差矩阵来准确表征神经网络的输入 - 输出分布。我们的技术的一个关键因素是解析解，用于描述通过非线性激活函数（如 Heaviside、ReLU 和 GELU）传递的随机变量的协方差。在对已训练的神经网络和贝叶斯神经网络的输入 - 输出分布进行分析的实验证明了所提出技术的广泛适用性和优点。

神经网络中协方差传播的解析解

An Analytic Solution to Covariance Propagation in Neural Networks

Deep neural networks (NNs) are known for their high-prediction performances.
However, NNs are prone to yield unreliable predictions when encountering
completely new situations without indicating their uncertainty. Bayesian
variants of NNs (BNNs), such as Monte Carlo (MC) dropout BNNs, do provide
uncertainty measures and simultaneously increase the prediction performance.
The only disadvantage of BNNs is their higher computation time during test time
because they rely on a sampling approach. Here we present a single-shot MC
dropout approximation that preserves the advantages of BNNs while being as fast
as NNs. Our approach is based on moment propagation (MP) and allows to
analytically approximate the expected value and the variance of the MC dropout
signal for commonly used layers in NNs, i.e. convolution, max pooling, dense,
softmax, and dropout layers. The MP approach can convert an NN into a BNN
without re-training given the NN has been trained with standard dropout. We
evaluate our approach on different benchmark datasets and a simulated toy
example in a classification and regression setting. We demonstrate that our
single-shot MC dropout approximation resembles the point estimate and the
uncertainty estimate of the predictive distribution that is achieved with an MC
approach, while being fast enough for real-time deployments of BNNs. We show
that using part of the saved time to combine our MP approach with deep ensemble
techniques does further improve the uncertainty measures.

我们提出了一种快速的基于矩阵传播的近似方法，将神经网络转换成贝叶斯神经网络，实现对预测分布的点估计和不确定性估计，并通过与深度集成技术相结合进一步提高不确定性测量的性能。