Recent work in deep learning has shown strong empirical and theoretical
evidence of an implicit low-rank bias: weight matrices in deep networks tend to
be approximately low-rank and removing relatively small singular values during
training or from available trained models may significantly reduce model size
while maintaining or even improving model performance. However, the majority of
the theoretical investigations around low-rank bias in neural networks deal
with oversimplified deep linear networks. In this work, we consider general
networks with nonlinear activations and the weight decay parameter, and we show
the presence of an intriguing neural rank collapse phenomenon, connecting the
low-rank bias of trained networks with networks' neural collapse properties: as
the weight decay parameter grows, the rank of each layer in the network
decreases proportionally to the within-class variability of the hidden-space
embeddings of the previous layers. Our theoretical findings are supported by a
range of experimental evaluations illustrating the phenomenon.

深度学习中的低秩偏好与神经网络的神经层塌陷现象相关，权重衰减参数的增长导致网络中每一层的秩与前一层隐藏空间嵌入的类内变异成正比减少。