The scalability of deep learning models is fundamentally limited by computing
resources, memory, and communication. Although methods like low-rank adaptation
(LoRA) have reduced the cost of model finetuning, its application in model
pre-training remains largely unexplored. This paper explores extending LoRA to
model pre-training, identifying the inherent constraints and limitations of
standard LoRA in this context. We introduce LoRA-the-Explorer (LTE), a novel
bi-level optimization algorithm designed to enable parallel training of
multiple low-rank heads across computing nodes, thereby reducing the need for
frequent synchronization. Our approach includes extensive experimentation on
vision transformers using various vision datasets, demonstrating that LTE is
competitive with standard pre-training.

深度学习模型的可扩展性受到计算资源、内存和通讯的根本限制。本文在模型预训练中探索了低秩适应（LoRA）方法的应用，介绍了 LoRA-the-Explorer（LTE）算法，通过在计算节点上进行多个低秩头的并行训练来减少同步频率，使用不同视觉数据集进行广泛实验，并证明 LTE 在标准预训练中具有竞争力。