Selecting hyperparameters in deep learning greatly impacts its effectiveness
but requires manual effort and expertise. Recent works show that Bayesian model
selection with Laplace approximations can allow to optimize such
hyperparameters just like standard neural network parameters using gradients
and on the training data. However, estimating a single hyperparameter gradient
requires a pass through the entire dataset, limiting the scalability of such
algorithms. In this work, we overcome this issue by introducing lower bounds to
the linearized Laplace approximation of the marginal likelihood. In contrast to
previous estimators, these bounds are amenable to stochastic-gradient-based
optimization and allow to trade off estimation accuracy against computational
complexity. We derive them using the function-space form of the linearized
Laplace, which can be estimated using the neural tangent kernel.
Experimentally, we show that the estimators can significantly accelerate
gradient-based hyperparameter optimization.

本研究介绍了一种基于贝叶斯模型选择和拉普拉斯逼近的方法，通过引入下界到边缘似然的线性化拉普拉斯逼近，用于选择深度学习的超参数优化，该方法可以使用随机梯度基于优化，并可以利用神经切向核估计。实验结果表明，该估计器可以显着加速基于梯度的超参数优化。