Hyperparameter selection generally relies on running multiple full training
trials, with selection based on validation set performance. We propose a
gradient-based approach for locally adjusting hyperparameters during training
of the model. Hyperparameters are adjusted so as to make the model parameter
gradients, and hence updates, more advantageous for the validation cost. We
explore the approach for tuning regularization hyperparameters and find that in
experiments on MNIST, SVHN and CIFAR-10, the resulting regularization levels
are within the optimal regions. The additional computational cost depends on
how frequently the hyperparameters are trained, but the tested scheme adds only
30% computational overhead regardless of the model size. Since the method is
significantly less computationally demanding compared to similar gradient-based
approaches to hyperparameter optimization, and consistently finds good
hyperparameter values, it can be a useful tool for training neural network
models.

本文提出一种基于梯度的方法来调整模型的超参数，使其在对验证成本更有利的情况下进行模型参数梯度和更新，实现对正则化超参数的调优。在 MNIST、SVHN 和 CIFAR-10 数据集上的实验表明，此方法比其他基于梯度的方法成本更低且一致找到了好的超参数值，有望成为神经网络模型训练的有用工具。

基于梯度的可扩展连续正则化超参数调整

Scalable Gradient-Based Tuning of Continuous Regularization  Hyperparameters

To improve the efficiency of Monte Carlo estimation, practitioners are
turning to biased Markov chain Monte Carlo procedures that trade off asymptotic
exactness for computational speed. The reasoning is sound: a reduction in
variance due to more rapid sampling can outweigh the bias introduced. However,
the inexactness creates new challenges for sampler and parameter selection,
since standard measures of sample quality like effective sample size do not
account for asymptotic bias. To address these challenges, we introduce a new
computable quality measure based on Stein's method that quantifies the maximum
discrepancy between sample and target expectations over a large class of test
functions. We use our tool to compare exact, biased, and deterministic sample
sequences and illustrate applications to hyperparameter selection, convergence
rate assessment, and quantifying bias-variance tradeoffs in posterior
inference.

为了提高 Monte Carlo 估计的效率，研究者们正转向有偏的马尔可夫链蒙特卡罗过程，通过权衡渐近精确度和计算速度来实现。本文引入一种基于 Stein's 方法的可计算质量度量来解决这些配合中不精确性带来的新挑战，并将其应用于超参数选择、收敛速率评估和后验推断中，比较精确、有偏和确定性样本序列，并量化样本和目标期望之间的最大偏差。

使用 Stein 方法测量样本质量

Measuring Sample Quality with Stein's Method

One of the most tedious tasks in the application of machine learning is model
selection, i.e. hyperparameter selection. Fortunately, recent progress has been
made in the automation of this process, through the use of sequential
model-based optimization (SMBO) methods. This can be used to optimize a
cross-validation performance of a learning algorithm over the value of its
hyperparameters. However, it is well known that ensembles of learned models
almost consistently outperform a single model, even if properly selected. In
this paper, we thus propose an extension of SMBO methods that automatically
constructs such ensembles. This method builds on a recently proposed ensemble
construction paradigm known as agnostic Bayesian learning. In experiments on 22
regression and 39 classification data sets, we confirm the success of this
proposed approach, which is able to outperform model selection with SMBO.

本文提出了一种新的基于 SMBO 和贝叶斯学习的自动构建学习模型合集的方法，实验证明其优于传统的 SMBO 方法在回归以及分类数据集上的表现。