Instruction Tuning has the potential to stimulate or enhance specific
capabilities of large language models (LLMs). However, achieving the right
balance of data is crucial to prevent catastrophic forgetting and interference
between tasks. To address these limitations and enhance training flexibility,
we propose the Mixture-of-LoRAs (MoA) architecture which is a novel and
parameter-efficient tuning method designed for multi-task learning with LLMs.
In this paper, we start by individually training multiple domain-specific LoRA
modules using corresponding supervised corpus data. These LoRA modules can be
aligned with the expert design principles observed in Mixture-of-Experts (MoE).
Subsequently, we combine the multiple LoRAs using an explicit routing strategy
and introduce domain labels to facilitate multi-task learning, which help
prevent interference between tasks and ultimately enhances the performance of
each individual task. Furthermore, each LoRA model can be iteratively adapted
to a new domain, allowing for quick domain-specific adaptation. Experiments on
diverse tasks demonstrate superior and robust performance, which can further
promote the wide application of domain-specific LLMs.

使用 Mixture-of-LoRAs (MoA) 架构的多任务学习方法，通过培训多个领域特定的 LoRA 模块，采用明确的路由策略和领域标签来防止任务之间的干扰，并最终提高每个个体任务的性能，进而促进领域特定大语言模型（LLMs）的广泛应用。