With the rapid growth in the scale of pre-trained foundation models,
parameter-efficient fine-tuning techniques have gained significant attention,
among which Adapter Tuning is the most widely used. Despite achieving
efficiency, Adapter Tuning still underperforms full fine-tuning, and the
performance improves at the cost of an increase in parameters. Recent efforts
address this issue by pruning the original adapters, but it also introduces
training instability and suboptimal performance on certain datasets. Motivated
by this, we propose Mixture of Sparse Adapters, or MoSA, as a novel Adapter
Tuning method to fully unleash the potential of each parameter in the adapter.
We first split the standard adapter into multiple non-overlapping modules, then
stochastically activate modules for sparse training, and finally merge them to
form a complete adapter after tuning. In this way, MoSA can achieve
significantly better performance than standard adapters without any additional
computational or storage overhead. Furthermore, we propose a hierarchical
sparse strategy to better leverage limited training data. Extensive experiments
on a series of 27 visual tasks demonstrate that MoSA consistently outperforms
other Adapter Tuning methods as well as other baselines by a significant
margin. Furthermore, in two challenging scenarios with low-resource and
multi-task settings, MoSA achieves satisfactory results, further demonstrating
the effectiveness of our design. Our code will be released.

通过提出混合稀疏适配器（MoSA）作为一种新颖的适配器调整方法，旨在充分发挥每个适配器参数的潜力，本研究对一系列 27 个视觉任务进行了广泛实验证明，MoSA 始终比其他适配器调整方法和基准线方法表现更好，并在低资源和多任务设置的两个具有挑战性的场景中展现了令人满意的结果。