In the era of large language models, the demand for efficient use of
computational resources has become critically important. Although
parameter-efficient fine-tuning techniques have achieved results comparable to
full fine-tuning, their application during the pre-training phase poses
significant challenges. Specifically, employing parameter-efficient strategies
at the onset of pre-training can severely compromise efficiency, especially in
larger models. In this paper, building upon the fine-tuning method LoRA, we
introduce a novel parameter-efficient training technique that frequently alters
trainable part of parameters, facilitating effective pre-training. Our method
not only achieves memory reductions and computational overhead comparable to
current state-of-the-art parameter-efficient algorithms during the pre-training
phase but also maintains accuracy levels comparable to those of full
pre-training. We provide both theoretical analyses and empirical evidence to
demonstrate the effectiveness of our approach.

在大语言模型时代，提高计算资源的有效利用需求变得非常重要。本文基于 LoRA 精调方法，引入了一种新颖的参数高效训练技术，通过频繁改变可训练参数的一部分，提高了有效的预训练。我们的方法不仅在预训练阶段实现了内存和计算开销减少，与当前最先进的参数高效算法相当，而且保持了与完全预训练相当的准确性水平。我们提供了理论分析和实证证据来证明我们的方法的有效性。

通过动态参数调整彻底改变大型语言模型训练

Revolutionizing Large Language Model Training through Dynamic Parameter  Adjustment

With the rapid growth in the scale of pre-trained foundation models,
parameter-efficient fine-tuning techniques have gained significant attention,
among which Adapter Tuning is the most widely used. Despite achieving
efficiency, Adapter Tuning still underperforms full fine-tuning, and the
performance improves at the cost of an increase in parameters. Recent efforts
address this issue by pruning the original adapters, but it also introduces
training instability and suboptimal performance on certain datasets. Motivated
by this, we propose Mixture of Sparse Adapters, or MoSA, as a novel Adapter
Tuning method to fully unleash the potential of each parameter in the adapter.
We first split the standard adapter into multiple non-overlapping modules, then
stochastically activate modules for sparse training, and finally merge them to
form a complete adapter after tuning. In this way, MoSA can achieve
significantly better performance than standard adapters without any additional
computational or storage overhead. Furthermore, we propose a hierarchical
sparse strategy to better leverage limited training data. Extensive experiments
on a series of 27 visual tasks demonstrate that MoSA consistently outperforms
other Adapter Tuning methods as well as other baselines by a significant
margin. Furthermore, in two challenging scenarios with low-resource and
multi-task settings, MoSA achieves satisfactory results, further demonstrating
the effectiveness of our design. Our code will be released.

通过提出混合稀疏适配器（MoSA）作为一种新颖的适配器调整方法，旨在充分发挥每个适配器参数的潜力，本研究对一系列 27 个视觉任务进行了广泛实验证明，MoSA 始终比其他适配器调整方法和基准线方法表现更好，并在低资源和多任务设置的两个具有挑战性的场景中展现了令人满意的结果。