Our understanding of learning dynamics of deep neural networks (DNNs) remains incomplete. Recent research has begun to uncover the mathematical principles underlying these networks, including the phenomenon of "Neural Collapse", where linear classifiers within DNNs converge to specific geometrical structures during late-stage training. However, the role of geometric constraints in learning extends beyond this terminal phase. For instance, gradients in fully-connected layers naturally develop a low-rank structure due to the accumulation of rank-one outer products over a training batch. Despite the attention given to methods that exploit this structure for memory saving or regularization, the emergence of low-rank learning as an inherent aspect of certain DNN architectures has been under-explored. In this paper, we conduct a comprehensive study of gradient rank in DNNs, examining how architectural choices and structure of the data effect gradient rank bounds. Our theoretical analysis provides these bounds for training fully-connected, recurrent, and convolutional neural networks. We also demonstrate, both theoretically and empirically, how design choices like activation function linearity, bottleneck layer introduction, convolutional stride, and sequence truncation influence these bounds. Our findings not only contribute to the understanding of learning dynamics in DNNs, but also provide practical guidance for deep learning engineers to make informed design decisions.

我们在深度神经网络的学习动态方面的理解仍然不完整。最近的研究开始揭示了这些网络的数学原理，包括“神经坍塌”现象，在训练的后期，DNN内的线性分类器会收敛到特定的几何结构。然而，几何约束在学习中的作用并不仅限于这个阶段。本文对DNN中的梯度秩进行了全面研究，研究了体系结构选择和数据结构对梯度秩界限的影响。我们的研究不仅有助于理解DNN中的学习动态，而且为深度学习工程师提供了实践指导，以便做出明智的设计决策。

低秩学习设计：网络架构和激活线性在梯度秩崩溃中的作用