Well-tuned hyperparameters are crucial for obtaining good generalization
behavior in neural networks. They can enforce appropriate inductive biases,
regularize the model and improve performance -- especially in the presence of
limited data. In this work, we propose a simple and efficient way for
optimizing hyperparameters inspired by the marginal likelihood, an optimization
objective that requires no validation data. Our method partitions the training
data and a neural network model into $K$ data shards and parameter partitions,
respectively. Each partition is associated with and optimized only on specific
data shards. Combining these partitions into subnetworks allows us to define
the ``out-of-training-sample" loss of a subnetwork, i.e., the loss on data
shards unseen by the subnetwork, as the objective for hyperparameter
optimization. We demonstrate that we can apply this objective to optimize a
variety of different hyperparameters in a single training run while being
significantly computationally cheaper than alternative methods aiming to
optimize the marginal likelihood for neural networks. Lastly, we also focus on
optimizing hyperparameters in federated learning, where retraining and
cross-validation are particularly challenging.

本篇论文提出了一种简单且高效的方法来优化神经网络的超参数，采用无需验证数据的优化目标 —— 边缘似然来实现，将训练数据和神经网络模型分别划分为 $K$ 个数据分片和参数分区，仅在特定的数据分片上关联并优化每个分区，然后将这些分区组合成子网络，通过所定义的子网络的 “训练外样本” 损失作为超参数优化的目标来优化多种不同类型的超参数。最后，该方法特别适用于在联邦学习中优化超参数的情况，其中重新训练和交叉验证尤为具有挑战性。