Overparameterized models have proven to be powerful tools for solving various
machine learning tasks. However, overparameterization often leads to a
substantial increase in computational and memory costs, which in turn requires
extensive resources to train. In this work, we aim to reduce this complexity by
studying the learning dynamics of overparameterized deep networks. By
extensively studying its learning dynamics, we unveil that the weight matrices
of various architectures exhibit a low-dimensional structure. This finding
implies that we can compress the networks by reducing the training to a small
subspace. We take a step in developing a principled approach for compressing
deep networks by studying deep linear models. We demonstrate that the principal
components of deep linear models are fitted incrementally but within a small
subspace, and use these insights to compress deep linear networks by decreasing
the width of its intermediate layers. Remarkably, we observe that with a
particular choice of initialization, the compressed network converges faster
than the original network, consistently yielding smaller recovery errors
throughout all iterations of gradient descent. We substantiate this observation
by developing a theory focused on the deep matrix factorization problem, and by
conducting empirical evaluations on deep matrix sensing. Finally, we
demonstrate how our compressed model can enhance the utility of deep nonlinear
models. Overall, we observe that our compression technique accelerates the
training process by more than 2x, without compromising model quality.

通过研究过度参数化的深度网络的学习动力学，我们揭示了各种体系结构的权重矩阵展现出低维结构，我们利用这些洞见通过减小中间层的宽度来压缩深度线性网络，实验证明这种压缩技术能够加速训练过程超过两倍，而不牺牲模型质量。

通过低维度学习动力学高效压缩超参数化深度模型

Efficient Compression of Overparameterized Deep Models through  Low-Dimensional Learning Dynamics

A leading hypothesis for the surprising generalization of neural networks is
that the dynamics of gradient descent bias the model towards simple solutions,
by searching through the solution space in an incremental order of complexity.
We formally define the notion of incremental learning dynamics and derive the
conditions on depth and initialization for which this phenomenon arises in deep
linear models. Our main theoretical contribution is a dynamical depth
separation result, proving that while shallow models can exhibit incremental
learning dynamics, they require the initialization to be exponentially small
for these dynamics to present themselves. However, once the model becomes
deeper, the dependence becomes polynomial and incremental learning can arise in
more natural settings. We complement our theoretical findings by experimenting
with deep matrix sensing, quadratic neural networks and with binary
classification using diagonal and convolutional linear networks, showing all of
these models exhibit incremental learning.

本文定义了增量学习动力学的概念并证明了在深度为多项式关系和初始化条件正确的情况下，神经网络可以展现出增量学习能力，且通过实验证明在使用深度学习模型中，梯度下降算法有助于寻找简单的模型解。