In this work, we study the evolution of the loss Hessian across many
classification tasks in order to understand the effect the curvature of the
loss has on the training dynamics. Whereas prior work has focused on how
different learning rates affect the loss Hessian observed during training, we
also analyze the effects of model initialization, architectural choices, and
common training heuristics such as gradient clipping and learning rate warmup.
Our results demonstrate that successful model and hyperparameter choices allow
the early optimization trajectory to either avoid -- or navigate out of --
regions of high curvature and into flatter regions that tolerate a higher
learning rate. Our results suggest a unifying perspective on how disparate
mitigation strategies for training instability ultimately address the same
underlying failure mode of neural network optimization, namely poor
conditioning. Inspired by the conditioning perspective, we show that learning
rate warmup can improve training stability just as much as batch normalization,
layer normalization, MetaInit, GradInit, and Fixup initialization.

本论文探究了损失海森矩阵在多项分类任务中的演化，以了解损失曲率对训练动态的影响。结果表明，成功的模型和超参数选择能够使早期的优化轨迹避免或穿越高曲率区域并进入扁平区域，提高学习率稳定性，类似于各种训练不稳定性缓解策略最终解决神经网络优化的相同失效模式，即差的条件。

深度学习训练不稳定性的损失曲率视角

A Loss Curvature Perspective on Training Instability in Deep Learning

Training heuristics greatly improve various image classification model
accuracies~\cite{he2018bag}. Object detection models, however, have more
complex neural network structures and optimization targets. The training
strategies and pipelines dramatically vary among different models. In this
works, we explore training tweaks that apply to various models including Faster
R-CNN and YOLOv3. These tweaks do not change the model architectures,
therefore, the inference costs remain the same. Our empirical results
demonstrate that, however, these freebies can improve up to 5% absolute
precision compared to state-of-the-art baselines.

通过使用一些 training heuristics，不改变模型结构，我们研究并在各种模型中应用 training tweaks，以提高物体检测的准确性。