While overparameterization in machine learning models offers great benefits
in terms of optimization and generalization, it also leads to increased
computational requirements as model sizes grow. In this work, we show that by
leveraging the inherent low-dimensional structures of data and compressible
dynamics within the model parameters, we can reap the benefits of
overparameterization without the computational burdens. In practice, we
demonstrate the effectiveness of this approach for deep low-rank matrix
completion as well as fine-tuning language models. Our approach is grounded in
theoretical findings for deep overparameterized low-rank matrix recovery, where
we show that the learning dynamics of each weight matrix are confined to an
invariant low-dimensional subspace. Consequently, we can construct and train
compact, highly compressed factorizations possessing the same benefits as their
overparameterized counterparts. In the context of deep matrix completion, our
technique substantially improves training efficiency while retaining the
advantages of overparameterization. For language model fine-tuning, we propose
a method called "Deep LoRA", which improves the existing low-rank adaptation
(LoRA) technique, leading to reduced overfitting and a simplified
hyperparameter setup, while maintaining comparable efficiency. We validate the
effectiveness of Deep LoRA on natural language tasks, particularly when
fine-tuning with limited data.

通过利用数据的固有低维结构和模型参数的可压缩动力学，我们展示了优化和泛化方面的超参数化的好处，而无需增加计算负担。在深度低秩矩阵补全和微调语言模型的实践中，我们证明了这种方法的有效性，同时保留了超参数化对性能的优势。