Bayesian neural networks address epistemic uncertainty by learning a
posterior distribution over model parameters. Sampling and weighting networks
according to this posterior yields an ensemble model referred to as Bayes
ensemble. Ensembles of neural networks (deep ensembles) can profit from the
cancellation of errors effect: Errors by ensemble members may average out and
the deep ensemble achieves better predictive performance than each individual
network. We argue that neither the sampling nor the weighting in a Bayes
ensemble are particularly well-suited for increasing generalization
performance, as they do not support the cancellation of errors effect, which is
evident in the limit from the Bernstein-von~Mises theorem for misspecified
models. In contrast, a weighted average of models where the weights are
optimized by minimizing a PAC-Bayesian generalization bound can improve
generalization performance. This requires that the optimization takes
correlations between models into account, which can be achieved by minimizing
the tandem loss at the cost that hold-out data for estimating error
correlations need to be available. The PAC-Bayesian weighting increases the
robustness against correlated models and models with lower performance in an
ensemble. This allows us to safely add several models from the same learning
process to an ensemble, instead of using early-stopping for selecting a single
weight configuration. Our study presents empirical results supporting these
conceptual considerations on four different classification datasets. We show
that state-of-the-art Bayes ensembles from the literature, despite being
computationally demanding, do not improve over simple uniformly weighted deep
ensembles and cannot match the performance of deep ensembles weighted by
optimizing the tandem loss, which additionally come with non-vacuous
generalization guarantees.

贝叶斯神经网络通过学习模型参数的后验分布来解决认知不确定性问题。使用该后验进行抽样和加权网络，从而形成一个被称为贝叶斯集合的集合模型。相对于个别网络，深度集合可以受益于误差抵消效应，提高预测性能。本文论证了贝叶斯集合的抽样和加权方法并不适合增加泛化性能，因为它们无法实现误差抵消效应。相反，通过优化 PAC-Bayesian 泛化约束得到模型的加权平均值可以提高泛化性能。这种加权方法需要考虑模型之间的相关性，可以通过最小化串级损失来实现。PAC-Bayesian 加权方法增强了对相关模型和性能较低模型的稳健性，因此我们可以安全地添加来自同一学习过程的多个模型到集合中，而不是使用早停法选择单一的权重配置。本研究在四个不同的分类数据集上给出了实证结果，表明尽管计算代价较高，但来自文献中的最先进的贝叶斯集合并没有改进普通权重的深度集合，并且无法与通过优化串级损失的深度集合相匹配，后者还具有非空泛化保证。

贝叶斯与 PAC-Bayesian 深度神经网络集成

Bayesian vs. PAC-Bayesian Deep Neural Network Ensembles

We introduce a novel strategy to train randomised predictors in federated
learning, where each node of the network aims at preserving its privacy by
releasing a local predictor but keeping secret its training dataset with
respect to the other nodes. We then build a global randomised predictor which
inherits the properties of the local private predictors in the sense of a
PAC-Bayesian generalisation bound. We consider the synchronous case where all
nodes share the same training objective (derived from a generalisation bound),
and the asynchronous case where each node may have its own personalised
training objective. We show through a series of numerical experiments that our
approach achieves a comparable predictive performance to that of the batch
approach where all datasets are shared across nodes. Moreover the predictors
are supported by numerically nonvacuous generalisation bounds while preserving
privacy for each node. We explicitly compute the increment on predictive
performance and generalisation bounds between batch and federated settings,
highlighting the price to pay to preserve privacy.

我们提出了一种在联邦学习中训练随机预测器的新策略，其中网络的每个节点通过发布本地预测器并保密其训练数据集来保护隐私。我们构建了一个全局随机预测器，它在 PAC-Bayesian 广义界限的意义下继承了本地私有预测器的属性。我们考虑同步情况，所有节点共享相同的训练目标（来自广义界限）以及异步情况，每个节点可以拥有自己的个性化训练目标。通过一系列的数值实验，我们证明了我们的方法在预测性能方面与批处理方法相当，并且在保护每个节点隐私的同时具有数值上实质性的广义界限。我们明确计算了批处理和联邦学习设置之间的预测性能和广义界限的增量，突出了保护隐私的代价。