Curriculum learning (CL) posits that machine learning models -- similar to
humans -- may learn more efficiently from data that match their current
learning progress. However, CL methods are still poorly understood and, in
particular for natural language processing (NLP), have achieved only limited
success. In this paper, we explore why. Starting from an attempt to replicate
and extend a number of recent curriculum methods, we find that their results
are surprisingly brittle when applied to NLP. A deep dive into the
(in)effectiveness of the curricula in some scenarios shows us why: when
curricula are employed in combination with the popular Adam optimisation
algorithm, they oftentimes learn to adapt to suboptimally chosen optimisation
parameters for this algorithm. We present a number of different case studies
with different common hand-crafted and automated CL approaches to illustrate
this phenomenon, and we find that none of them outperforms optimisation with
only Adam with well-chosen hyperparameters. As such, our results contribute to
understanding why CL methods work, but at the same time urge caution when
claiming positive results.

通过对最近的课程学习方法进行复制与扩展，研究发现在自然语言处理领域中，当课程与流行的 Adam 优化算法结合时，它们常常学习如何适应选择不佳的优化参数，从而导致结果异常脆弱，进而说明了为何课程学习方法在该领域的效果有限。

Adam 优化算法下的课程学习：细节的魔鬼存在

Curriculum Learning with Adam: The Devil Is in the Wrong Details

It is widely recognized that the generalization ability of neural networks
can be greatly enhanced through carefully designing the training procedure. The
current state-of-the-art training approach involves utilizing stochastic
gradient descent (SGD) or Adam optimization algorithms along with a combination
of additional regularization techniques such as weight decay, dropout, or noise
injection. Optimal generalization can only be achieved by tuning a multitude of
hyperparameters through grid search, which can be time-consuming and
necessitates additional validation datasets. To address this issue, we
introduce a practical PAC-Bayes training framework that is nearly tuning-free
and requires no additional regularization while achieving comparable testing
performance to that of SGD/Adam after a complete grid search and with extra
regularizations. Our proposed algorithm demonstrates the remarkable potential
of PAC training to achieve state-of-the-art performance on deep neural networks
with enhanced robustness and interpretability.

提出一种实用的 PAC-Bayes 训练框架，通过使用无需多重超参数调整的算法，结合 Stochastic Gradient Descent（SGD）或 Adam 优化算法和正则化技术等方法，实现了与常规方法相当的测试性能，同时实现了深度神经网络的鲁棒性和可解释性。

Auto-tune: 基于先验和后验的 PAC-Bayes 优化神经网络

Auto-tune: PAC-Bayes Optimization over Prior and Posterior for Neural  Networks

When training neural models, it is common to combine multiple loss terms. The
balancing of these terms requires considerable human effort and is
computationally demanding. Moreover, the optimal trade-off between the loss
term can change as training progresses, especially for adversarial terms. In
this work, we generalize the Adam optimization algorithm to handle multiple
loss terms. The guiding principle is that for every layer, the gradient
magnitude of the terms should be balanced. To this end, the Multi-Term Adam
(MTAdam) computes the derivative of each loss term separately, infers the first
and second moments per parameter and loss term, and calculates a first moment
for the magnitude per layer of the gradients arising from each loss. This
magnitude is used to continuously balance the gradients across all layers, in a
manner that both varies from one layer to the next and dynamically changes over
time. Our results show that training with the new method leads to fast recovery
from suboptimal initial loss weighting and to training outcomes that match
conventional training with the prescribed hyperparameters of each method.

本文介绍了一种使用 MTAdam 优化算法来平衡神经模型训练中的多个损失项的方法，使用这种方法可以可以在所有层次上平衡梯度大小，快速恢复从次优初始损失权重开始的训练结果，并且得出的训练结果符合传统方法每种超参数的规定。