Empirical and theoretical works show that the input weights of two-layer
neural networks, when initialized with small values, converge towards isolated
orientations. This phenomenon, referred to as condensation, indicates that the
gradient descent methods tend to spontaneously reduce the complexity of neural
networks during the training process. In this work, we elucidate the mechanisms
behind the condensation phenomena occurring in the training of three-layer
neural networks and distinguish it from the training of two-layer neural
networks. Through rigorous theoretical analysis, we establish the blow-up
property of effective dynamics and present a sufficient condition for the
occurrence of condensation, findings that are substantiated by experimental
results. Additionally, we explore the association between condensation and the
low-rank bias observed in deep matrix factorization.

通过理论分析，我们揭示了三层神经网络训练中凝聚现象的机制，并从二层神经网络训练中进行了区分。我们还建立了有效动力学的爆炸特性，并给出了凝聚现象发生的充分条件，并通过实验证实了这些发现。此外，我们探索了凝聚与深度矩阵分解中观察到的低秩偏差之间的关联。

三层神经网络动力学：初始凝聚

On the dynamics of three-layer neural networks: initial condensation

Recent work in deep learning has shown strong empirical and theoretical
evidence of an implicit low-rank bias: weight matrices in deep networks tend to
be approximately low-rank and removing relatively small singular values during
training or from available trained models may significantly reduce model size
while maintaining or even improving model performance. However, the majority of
the theoretical investigations around low-rank bias in neural networks deal
with oversimplified deep linear networks. In this work, we consider general
networks with nonlinear activations and the weight decay parameter, and we show
the presence of an intriguing neural rank collapse phenomenon, connecting the
low-rank bias of trained networks with networks' neural collapse properties: as
the weight decay parameter grows, the rank of each layer in the network
decreases proportionally to the within-class variability of the hidden-space
embeddings of the previous layers. Our theoretical findings are supported by a
range of experimental evaluations illustrating the phenomenon.

深度学习中的低秩偏好与神经网络的神经层塌陷现象相关，权重衰减参数的增长导致网络中每一层的秩与前一层隐藏空间嵌入的类内变异成正比减少。