Multiple techniques for producing calibrated predictive probabilities using deep neural networks in supervised learning settings have emerged that leverage approaches to ensemble diverse solutions discovered during cyclic training or training from multiple random starting points (deep ensembles). However, only a limited amount of work has investigated the utility of exploring the local region around each diverse solution (posterior mode). Using three well-known deep architectures on the CIFAR-10 dataset, we evaluate several simple methods for exploring local regions of the weight space with respect to Brier score, accuracy, and expected calibration error. We consider both Bayesian inference techniques (variational inference and Hamiltonian Monte Carlo applied to the softmax output layer) as well as utilizing the stochastic gradient descent trajectory near optima. While adding separate modes to the ensemble uniformly improves performance, we show that the simple mode exploration methods considered here produce little to no improvement over ensembles without mode exploration.

本文针对采用多个不同起点训练或在训练期间得到多个解法的深度集成方法，在探索与其算法解法的局部区域(后验模式)相比，评估了利用贝叶斯推断技术、Hamiltonian Monte Carlo等方法探索权重空间局部区域的效果，并在多个简单方法的组合中展示它们极少改进模式集成的性能。

贝叶斯模型平均中模式探索在神经网络中的有效性研究