Our understanding of learning dynamics of deep neural networks (DNNs) remains
incomplete. Recent research has begun to uncover the mathematical principles
underlying these networks, including the phenomenon of "Neural Collapse", where
linear classifiers within DNNs converge to specific geometrical structures
during late-stage training. However, the role of geometric constraints in
learning extends beyond this terminal phase. For instance, gradients in
fully-connected layers naturally develop a low-rank structure due to the
accumulation of rank-one outer products over a training batch. Despite the
attention given to methods that exploit this structure for memory saving or
regularization, the emergence of low-rank learning as an inherent aspect of
certain DNN architectures has been under-explored. In this paper, we conduct a
comprehensive study of gradient rank in DNNs, examining how architectural
choices and structure of the data effect gradient rank bounds. Our theoretical
analysis provides these bounds for training fully-connected, recurrent, and
convolutional neural networks. We also demonstrate, both theoretically and
empirically, how design choices like activation function linearity, bottleneck
layer introduction, convolutional stride, and sequence truncation influence
these bounds. Our findings not only contribute to the understanding of learning
dynamics in DNNs, but also provide practical guidance for deep learning
engineers to make informed design decisions.

我们在深度神经网络的学习动态方面的理解仍然不完整。最近的研究开始揭示了这些网络的数学原理，包括 “神经坍塌” 现象，在训练的后期，DNN 内的线性分类器会收敛到特定的几何结构。然而，几何约束在学习中的作用并不仅限于这个阶段。本文对 DNN 中的梯度秩进行了全面研究，研究了体系结构选择和数据结构对梯度秩界限的影响。我们的研究不仅有助于理解 DNN 中的学习动态，而且为深度学习工程师提供了实践指导，以便做出明智的设计决策。