Parameter-space regularization in neural network optimization is a
fundamental tool for improving generalization. However, standard
parameter-space regularization methods make it challenging to encode explicit
preferences about desired predictive functions into neural network training. In
this work, we approach regularization in neural networks from a probabilistic
perspective and show that by viewing parameter-space regularization as
specifying an empirical prior distribution over the model parameters, we can
derive a probabilistically well-motivated regularization technique that allows
explicitly encoding information about desired predictive functions into neural
network training. This method -- which we refer to as function-space empirical
Bayes (FSEB) -- includes both parameter- and function-space regularization, is
mathematically simple, easy to implement, and incurs only minimal computational
overhead compared to standard regularization techniques. We evaluate the
utility of this regularization technique empirically and demonstrate that the
proposed method leads to near-perfect semantic shift detection,
highly-calibrated predictive uncertainty estimates, successful task adaption
from pre-trained models, and improved generalization under covariate shift.

神经网络优化中的参数空间正则化是提高泛化能力的基本工具，但是标准的参数空间正则化方法使得将对期望预测函数的显式偏好编码到神经网络训练中变得具有挑战性。本文从概率的角度来看待神经网络的正则化，并展示了通过将参数空间正则化视为对模型参数的经验先验分布的规定，我们可以推导出一种概率上合理的正则化技术，使得将有关对期望预测函数的信息明确编码到神经网络训练中成为可能。这种方法被称为函数空间经验贝叶斯（FSEB），包括参数空间和函数空间的正则化，数学上简单、易于实现，并且与标准正则化技术相比，仅产生很小的计算开销。我们通过实证评估这种正则化技术的效用并证明了所提出的方法在语义转移检测、高度校准的预测不确定性估计、从预训练模型进行任务适应和在协变量转移下改进泛化能力等方面具有接近完美的性能。