LoRA has gained widespread acceptance in the fine-tuning of large pre-trained
models to cater to a diverse array of downstream tasks, showcasing notable
effectiveness and efficiency, thereby solidifying its position as one of the
most prevalent fine-tuning techniques. Due to the modular nature of LoRA's
plug-and-play plugins, researchers have delved into the amalgamation of
multiple LoRAs to empower models to excel across various downstream tasks.
Nonetheless, extant approaches for LoRA fusion grapple with inherent
challenges. Direct arithmetic merging may result in the loss of the original
pre-trained model's generative capabilities or the distinct identity of LoRAs,
thereby yielding suboptimal outcomes. On the other hand, Reference tuning-based
fusion exhibits limitations concerning the requisite flexibility for the
effective combination of multiple LoRAs. In response to these challenges, this
paper introduces the Mixture of LoRA Experts (MoLE) approach, which harnesses
hierarchical control and unfettered branch selection. The MoLE approach not
only achieves superior LoRA fusion performance in comparison to direct
arithmetic merging but also retains the crucial flexibility for combining LoRAs
effectively. Extensive experimental evaluations conducted in both the Natural
Language Processing (NLP) and Vision & Language (V&L) domains substantiate the
efficacy of MoLE.

引入了 Mixture of LoRA Experts (MoLE) 方法，利用分层控制和无限制的分支选择，实现了对 LoRA 的优化融合性能和弹性组合能力的提升。通过在自然语言处理（NLP）和视觉与语言（V&L）领域进行广泛的实验评估，证实了 MoLE 的有效性。