Bayesian neural networks address epistemic uncertainty by learning a
posterior distribution over model parameters. Sampling and weighting networks
according to this posterior yields an ensemble model referred to as Bayes
ensemble. Ensembles of neural networks (deep ensembles) can profit from the
cancellation of errors effect: Errors by ensemble members may average out and
the deep ensemble achieves better predictive performance than each individual
network. We argue that neither the sampling nor the weighting in a Bayes
ensemble are particularly well-suited for increasing generalization
performance, as they do not support the cancellation of errors effect, which is
evident in the limit from the Bernstein-von~Mises theorem for misspecified
models. In contrast, a weighted average of models where the weights are
optimized by minimizing a PAC-Bayesian generalization bound can improve
generalization performance. This requires that the optimization takes
correlations between models into account, which can be achieved by minimizing
the tandem loss at the cost that hold-out data for estimating error
correlations need to be available. The PAC-Bayesian weighting increases the
robustness against correlated models and models with lower performance in an
ensemble. This allows us to safely add several models from the same learning
process to an ensemble, instead of using early-stopping for selecting a single
weight configuration. Our study presents empirical results supporting these
conceptual considerations on four different classification datasets. We show
that state-of-the-art Bayes ensembles from the literature, despite being
computationally demanding, do not improve over simple uniformly weighted deep
ensembles and cannot match the performance of deep ensembles weighted by
optimizing the tandem loss, which additionally come with non-vacuous
generalization guarantees.

贝叶斯神经网络通过学习模型参数的后验分布来解决认知不确定性问题。使用该后验进行抽样和加权网络，从而形成一个被称为贝叶斯集合的集合模型。相对于个别网络，深度集合可以受益于误差抵消效应，提高预测性能。本文论证了贝叶斯集合的抽样和加权方法并不适合增加泛化性能，因为它们无法实现误差抵消效应。相反，通过优化 PAC-Bayesian 泛化约束得到模型的加权平均值可以提高泛化性能。这种加权方法需要考虑模型之间的相关性，可以通过最小化串级损失来实现。PAC-Bayesian 加权方法增强了对相关模型和性能较低模型的稳健性，因此我们可以安全地添加来自同一学习过程的多个模型到集合中，而不是使用早停法选择单一的权重配置。本研究在四个不同的分类数据集上给出了实证结果，表明尽管计算代价较高，但来自文献中的最先进的贝叶斯集合并没有改进普通权重的深度集合，并且无法与通过优化串级损失的深度集合相匹配，后者还具有非空泛化保证。