We study deep neural networks for the multi-label classification (MLab) task
through the lens of neural collapse (NC). Previous works have been restricted
to the multi-class classification setting and discovered a prevalent NC
phenomenon comprising of the following properties for the last-layer features:
(i) the variability of features within every class collapses to zero, (ii) the
set of feature means form an equi-angular tight frame (ETF), and (iii) the last
layer classifiers collapse to the feature mean upon some scaling. We generalize
the study to multi-label learning, and prove for the first time that a
generalized NC phenomenon holds with the "pick-all-label'' formulation. Under
the natural analog of the unconstrained feature model (UFM), we establish that
the only global classifier of the pick-all-label cross entropy loss display the
same ETF geometry which further collapse to multiplicity-1 feature class means.
Besides, we discover a combinatorial property in generalized NC which is unique
for multi-label learning that we call ``tag-wise average'' property, where the
feature class-means of samples with multiple labels are scaled average of the
feature class-means of single label tags. Theoretically, we establish global
optimality result for the pick-all-label cross-entropy risk for the UFM.
Additionally, We also provide empirical evidence to support our investigation
into training deep neural networks on multi-label datasets, resulting in
improved training efficiency.

我们研究了深度神经网络在多标签分类任务中的神经折叠现象，通过研究我们证明了具有 “选择全部标签” 形式的广义神经折叠现象成立，同时发现了多标签学习中独特的 “标签平均” 的组合属性，并在理论上建立了全局优化结果和训练效率的实证证据。