Adapting pretrained large language models (LLMs) to various downstream tasks
in tens or hundreds of human languages is computationally expensive.
Parameter-efficient fine-tuning (PEFT) significantly reduces the adaptation
cost, by tuning only a small amount of parameters. However, directly applying
PEFT methods such as LoRA (Hu et al., 2022) on diverse dataset mixtures could
lead to suboptimal performance due to limited parameter capacity and negative
interference among different datasets. In this work, we propose Featurized
Low-rank Mixtures (FLix), a novel PEFT method designed for effective multitask
multilingual tuning. FLix associates each unique dataset feature, such as the
dataset's language or task, with its own low-rank weight update parameters. By
composing feature-specific parameters for each dataset, FLix can accommodate
diverse dataset mixtures and generalize better to unseen datasets. Our
experiments show that FLix leads to significant improvements over a variety of
tasks for both supervised learning and zero-shot settings using different
training data mixtures.

针对多种多语言下游任务中，将预训练的大语言模型（LLM）进行调整以适应不同语言的计算成本过高的问题，本研究提出了一种参数高效的微调方法，即 Featurized Low-rank Mixtures（FLix），通过为每个数据集分配独立的低秩权重更新参数，实现了对多样的数据集混合的适应并对未见数据集表现更好。实验证明，FLix 在使用不同训练数据混合的有监督学习和零样本设置中，对多种任务都取得了显著的改进。