Recent works have studied implicit biases in deep learning, especially the
behavior of last-layer features and classifier weights. However, they usually
need to simplify the intermediate dynamics under gradient flow or gradient
descent due to the intractability of loss functions and model architectures. In
this paper, we introduce the unhinged loss, a concise loss function, that
offers more mathematical opportunities to analyze the closed-form dynamics
while requiring as few simplifications or assumptions as possible. The unhinged
loss allows for considering more practical techniques, such as time-vary
learning rates and feature normalization. Based on the layer-peeled model that
views last-layer features as free optimization variables, we conduct a thorough
analysis in the unconstrained, regularized, and spherical constrained cases, as
well as the case where the neural tangent kernel remains invariant. To bridge
the performance of the unhinged loss to that of Cross-Entropy (CE), we
investigate the scenario of fixing classifier weights with a specific
structure, (e.g., a simplex equiangular tight frame). Our analysis shows that
these dynamics converge exponentially fast to a solution depending on the
initialization of features and classifier weights. These theoretical results
not only offer valuable insights, including explicit feature regularization and
rescaled learning rates for enhancing practical training with the unhinged
loss, but also extend their applicability to other loss functions. Finally, we
empirically demonstrate these theoretical results and insights through
extensive experiments.

最近的研究重点研究了深度学习中的隐性偏差，特别是最后一层特征和分类器权重的行为。本文引入了简明的无约束损失函数，提供了更多数学机会来分析封闭动态，同时尽可能少地进行简化或假设。无约束损失允许考虑更多实际技术，如变化的学习率和特征归一化。通过将最后一层特征视为自由优化变量的层剥模型，我们在无约束、正则化、球面约束和神经切向核保持不变的情况下进行了彻底分析。我们研究了将无约束损失与交叉熵（CE）的性能联系起来的情况，其中分类器权重采用特定结构（如简单赫尔曼紧束框架）进行固定。我们的分析表明，这些动态按指数速度收敛到解，这取决于特征和分类器权重的初始化。这些理论结果不仅提供了有价值的见解，包括显式特征正则化和调整学习率以增强无约束损失实际训练的可行性，还将其适用性扩展到其他损失函数。最后，我们通过大量实验证明了这些理论结果和见解。

在脱轨损失及其之外的动力学下

On the Dynamics Under the Unhinged Loss and Beyond

Neural collapse provides an elegant mathematical characterization of learned
last layer representations (a.k.a. features) and classifier weights in deep
classification models. Such results not only provide insights but also motivate
new techniques for improving practical deep models. However, most of the
existing empirical and theoretical studies in neural collapse focus on the case
that the number of classes is small relative to the dimension of the feature
space. This paper extends neural collapse to cases where the number of classes
are much larger than the dimension of feature space, which broadly occur for
language models, retrieval systems, and face recognition applications. We show
that the features and classifier exhibit a generalized neural collapse
phenomenon, where the minimum one-vs-rest margins is maximized.We provide
empirical study to verify the occurrence of generalized neural collapse in
practical deep neural networks. Moreover, we provide theoretical study to show
that the generalized neural collapse provably occurs under unconstrained
feature model with spherical constraint, under certain technical conditions on
feature dimension and number of classes.

该论文扩展了神经崩溃理论到类别数量远大于特征空间维度的情况，并提供了实证和理论研究以验证广义神经崩溃现象的存在和原理。

大规模多类别问题的广义神经塌陷

Generalized Neural Collapse for a Large Number of Classes

Gait recognition is to seek correct matches for query individuals by their
unique walking patterns at a long distance. However, current methods focus
solely on individual gait features, disregarding inter-personal relationships.
In this paper, we reconsider gait representation, asserting that gait is not
just an aggregation of individual features, but also the relationships among
different subjects' gait features once reference gaits are established. From
this perspective, we redefine classifier weights as reference-anchored gaits,
allowing each person's gait to be described by their relationship with these
references. In our work, we call this novel descriptor Relationship Descriptor
(RD). This Relationship Descriptor offers two benefits: emphasizing meaningful
features and enhancing robustness. To be specific, The normalized dot product
between gait features and classifier weights signifies a similarity relation,
where each dimension indicates the similarity between the test sample and each
training ID's gait prototype, respectively. Despite its potential, the direct
use of relationship descriptors poses dimensionality challenges since the
dimension of RD depends on the training set's identity count. To address this,
we propose a Farthest Anchored gaits Selection algorithm and a dimension
reduction method to boost gait recognition performance. Our method can be built
on top of off-the-shelf pre-trained classification-based models without extra
parameters. We show that RD achieves higher recognition performance than
directly using extracted features. We evaluate the effectiveness of our method
on the popular GREW, Gait3D, CASIA-B, and OU-MVLP, showing that our method
consistently outperforms the baselines and achieves state-of-the-art
performances.

Gait recognition research using relationship descriptors to enhance recognition performance by considering inter-personal relationships among different subjects' gait features.