Adapting models pre-trained on large-scale datasets to a variety of
downstream tasks is a common strategy in deep learning. Consequently,
parameter-efficient fine-tuning methods have emerged as a promising way to
adapt pre-trained models to different tasks while training only a minimal
number of parameters. While most of these methods are designed for single-task
adaptation, parameter-efficient training in Multi-Task Learning (MTL)
architectures is still unexplored. In this paper, we introduce MTLoRA, a novel
framework for parameter-efficient training of MTL models. MTLoRA employs
Task-Agnostic and Task-Specific Low-Rank Adaptation modules, which effectively
disentangle the parameter space in MTL fine-tuning, thereby enabling the model
to adeptly handle both task specialization and interaction within MTL contexts.
We applied MTLoRA to hierarchical-transformer-based MTL architectures, adapting
them to multiple downstream dense prediction tasks. Our extensive experiments
on the PASCAL dataset show that MTLoRA achieves higher accuracy on downstream
tasks compared to fully fine-tuning the MTL model while reducing the number of
trainable parameters by 3.6x. Furthermore, MTLoRA establishes a Pareto-optimal
trade-off between the number of trainable parameters and the accuracy of the
downstream tasks, outperforming current state-of-the-art parameter-efficient
training methods in both accuracy and efficiency. Our code is publicly
available.

本文介绍了一种新的多任务学习（MTL）模型的参数有效训练框架 MTLoRA，该框架通过使用任务无关和任务特定的低秩适应模块，在 MTL 微调中有效地解离参数空间，从而使模型能够熟练处理 MTL 环境中的任务专业化和交互，并且在 PASCAL 数据集上的广泛实验表明，MTLoRA 在减少可训练参数数量的同时，比对 MTL 模型进行全面微调在下游任务上具有更高的准确性，同时在准确性和效率方面优于当前最先进的参数有效训练方法。