With the productive evolution of large language models (LLMs) in the field of
natural language processing (NLP), tons of effort has been made to effectively
fine-tune common pre-trained LLMs to fulfill a variety of tasks in one or
multiple specific domain. In practice, there are two prevailing ways, in which
the adaptation can be achieved: (i) Multiple Independent Models: Pre-trained
LLMs are fine-tuned a few times independently using the corresponding training
samples from each task. (ii) An Integrated Model: Samples from all tasks are
employed to fine-tune a pre-trianed LLM unitedly. To address the high computing
cost and seesawing issue simultaneously, we propose a unified framework that
implements a 1 + N mutli-task fine-tuning pattern in LLMs using a novel
Customized Gate Control (CGC) Low-rank Adaptation (LoRA) algorithm. Our work
aims to take an advantage of both MTL (i.e., CGC) and PEFT (i.e., LoRA) scheme.
For a given cluster of tasks, we design an innovative layer that contains two
types of experts as additional trainable parameters to make LoRA be compatible
with MTL. To comprehensively evaluate the proposed framework, we conduct
well-designed experiments on two public datasets. The experimental results
demonstrate that the unified framework with CGC-LoRA modules achieves higher
evaluation scores than all benchmarks on both two datasets.

通过使用一种新颖的 Customized Gate Control (CGC) Low-rank Adaptation (LoRA) 算法，我们提出了一个统一的框架，在大规模语言模型（LLMs）中实现了一种 1 + N 多任务微调模式，通过加入两种类型的专家，使 LoRA 与 MTL 相适应，并且在两个公共数据集上表现得比所有基准模型获得更高的评估分数。