Recent works have studied implicit biases in deep learning, especially the
behavior of last-layer features and classifier weights. However, they usually
need to simplify the intermediate dynamics under gradient flow or gradient
descent due to the intractability of loss functions and model architectures. In
this paper, we introduce the unhinged loss, a concise loss function, that
offers more mathematical opportunities to analyze the closed-form dynamics
while requiring as few simplifications or assumptions as possible. The unhinged
loss allows for considering more practical techniques, such as time-vary
learning rates and feature normalization. Based on the layer-peeled model that
views last-layer features as free optimization variables, we conduct a thorough
analysis in the unconstrained, regularized, and spherical constrained cases, as
well as the case where the neural tangent kernel remains invariant. To bridge
the performance of the unhinged loss to that of Cross-Entropy (CE), we
investigate the scenario of fixing classifier weights with a specific
structure, (e.g., a simplex equiangular tight frame). Our analysis shows that
these dynamics converge exponentially fast to a solution depending on the
initialization of features and classifier weights. These theoretical results
not only offer valuable insights, including explicit feature regularization and
rescaled learning rates for enhancing practical training with the unhinged
loss, but also extend their applicability to other loss functions. Finally, we
empirically demonstrate these theoretical results and insights through
extensive experiments.

最近的研究重点研究了深度学习中的隐性偏差，特别是最后一层特征和分类器权重的行为。本文引入了简明的无约束损失函数，提供了更多数学机会来分析封闭动态，同时尽可能少地进行简化或假设。无约束损失允许考虑更多实际技术，如变化的学习率和特征归一化。通过将最后一层特征视为自由优化变量的层剥模型，我们在无约束、正则化、球面约束和神经切向核保持不变的情况下进行了彻底分析。我们研究了将无约束损失与交叉熵（CE）的性能联系起来的情况，其中分类器权重采用特定结构（如简单赫尔曼紧束框架）进行固定。我们的分析表明，这些动态按指数速度收敛到解，这取决于特征和分类器权重的初始化。这些理论结果不仅提供了有价值的见解，包括显式特征正则化和调整学习率以增强无约束损失实际训练的可行性，还将其适用性扩展到其他损失函数。最后，我们通过大量实验证明了这些理论结果和见解。

在脱轨损失及其之外的动力学下

On the Dynamics Under the Unhinged Loss and Beyond

Convex potential minimisation is the de facto approach to binary
classification. However, Long and Servedio [2010] proved that under symmetric
label noise (SLN), minimisation of any convex potential over a linear function
class can result in classification performance equivalent to random guessing.
This ostensibly shows that convex losses are not SLN-robust. In this paper, we
propose a convex, classification-calibrated loss and prove that it is
SLN-robust. The loss avoids the Long and Servedio [2010] result by virtue of
being negatively unbounded. The loss is a modification of the hinge loss, where
one does not clamp at zero; hence, we call it the unhinged loss. We show that
the optimal unhinged solution is equivalent to that of a strongly regularised
SVM, and is the limiting solution for any convex potential; this implies that
strong l2 regularisation makes most standard learners SLN-robust. Experiments
confirm the SLN-robustness of the unhinged loss.

本研究提出了一种凸，分类校正损失并证明它是 SLN - 稳健的。该损失通过负无限来避免先前的结果，并且实验证明了 unhinged loss 的 SLN - 鲁棒性。