Marginal-likelihood based model-selection, even though promising, is rarely
used in deep learning due to estimation difficulties. Instead, most approaches
rely on validation data, which may not be readily available. In this work, we
present a scalable marginal-likelihood estimation method to select both
hyperparameters and network architectures, based on the training data alone.
Some hyperparameters can be estimated online during training, simplifying the
procedure. Our marginal-likelihood estimate is based on Laplace's method and
Gauss-Newton approximations to the Hessian, and it outperforms cross-validation
and manual-tuning on standard regression and image classification datasets,
especially in terms of calibration and out-of-distribution detection. Our work
shows that marginal likelihoods can improve generalization and be useful when
validation data is unavailable (e.g., in nonstationary settings).

作为深度学习中模型选择的有前途的边际似然方法由于参数估计上的困难很少被使用。本研究提出了可伸缩的边缘似然估计方法，用于基于训练数据独立地选择超参数和网络结构。该方法建立在拉普拉斯方法和高斯牛顿逼近的黑塞矩阵的基础上，并在标准回归和图像分类数据集上表现优异。此外，该方法还能够在缺少验证数据（例如在非平稳设置中）时提高一般化性能。

深度学习模型选择的可扩展边缘似然估计

Scalable Marginal Likelihood Estimation for Model Selection in Deep  Learning

Generalization in deep learning has been the topic of much recent theoretical
and empirical research. Here we introduce desiderata for techniques that
predict generalization errors for deep learning models in supervised learning.
Such predictions should 1) scale correctly with data complexity; 2) scale
correctly with training set size; 3) capture differences between architectures;
4) capture differences between optimization algorithms; 5) be quantitatively
not too far from the true error (in particular, be non-vacuous); 6) be
efficiently computable; and 7) be rigorous. We focus on generalization error
upper bounds, and introduce a categorisation of bounds depending on assumptions
on the algorithm and data. We review a wide range of existing approaches, from
classical VC dimension to recent PAC-Bayesian bounds, commenting on how well
they perform against the desiderata.
We next use a function-based picture to derive a marginal-likelihood
PAC-Bayesian bound. This bound is, by one definition, optimal up to a
multiplicative constant in the asymptotic limit of large training sets, as long
as the learning curve follows a power law, which is typically found in practice
for deep learning problems. Extensive empirical analysis demonstrates that our
marginal-likelihood PAC-Bayes bound fulfills desiderata 1-3 and 5. The results
for 6 and 7 are promising, but not yet fully conclusive, while only desideratum
4 is currently beyond the scope of our bound. Finally, we comment on why this
function-based bound performs significantly better than current parameter-based
PAC-Bayes bounds.

本研究提出了关于深度学习的泛化误差的准则，介绍了一种基于边际似然的 PAC-Bayesian Bound 方法来预测泛化误差，并进行了广泛实证分析以评估该方法的效果和特性。