The theory of greedy low-rank learning (GLRL) aims to explain the impressive
generalization capabilities of deep learning. It proves that stochastic
gradient-based training implicitly regularizes neural networks towards low-rank
solutions through a gradual increase of the rank during training. However,
there is a gap between theory and practice since GLRL requires an infinitesimal
initialization of the weights, which is not practical due to the fact that it
is a saddle point. In this work, we remove the assumption of infinitesimal
initialization by focusing on cumulative weight updates. We prove the
cumulative weight updates follow an incremental low-rank trajectory for
arbitrary orthogonal initialization of weights in a three-layer linear network.
Empirically, we demonstrate that our theory holds on a broad range of neural
networks (e.g., transformers) and standard training algorithms (e.g., SGD,
Adam). However, existing training algorithms do not exploit the low-rank
property to improve computational efficiency as the networks are not
parameterized in low-rank. To remedy this, we design a new training algorithm
Incremental Low-Rank Learning (InRank), which explicitly expresses cumulative
weight updates as low-rank matrices while incrementally augmenting their ranks
during training. We evaluate InRank on GPT-2, and our results indicate that
InRank achieves comparable prediction performance as the full-rank counterpart
while requiring at most 33% of the total ranks throughout training. We also
propose an efficient version of InRank that achieves a reduction of 20% in
total training time and 37% in memory usage when training GPT-medium on
WikiText-103 from scratch.

该研究提出了渐进式低秩学习技术，通过一个新的训练算法 InRank 来改进神经网络的训练，使得神经网络参数化后低秩化，以此提高计算效率并在 WikiText-103 上训练 GPT-Medium 模型时分别实现了 20% 的减少训练时间和 37% 的减少内存使用，同时保持了与全秩情况下相似的预测表现。