Large-scale pre-trained models have achieved remarkable success in various
computer vision tasks. A standard approach to leverage these models is to
fine-tune all model parameters for downstream tasks, which poses challenges in
terms of computational and storage costs. Recently, inspired by Natural
Language Processing (NLP), parameter-efficient transfer learning has been
successfully applied to vision tasks. However, most existing techniques
primarily focus on single-task adaptation, and despite limited research on
multi-task adaptation, these methods often exhibit suboptimal training and
inference efficiency. In this paper, we first propose an once-for-all Vision
Multi-Task Adapter (VMT-Adapter), which strikes approximately O(1) training and
inference efficiency w.r.t task number. Concretely, VMT-Adapter shares the
knowledge from multiple tasks to enhance cross-task interaction while preserves
task-specific knowledge via independent knowledge extraction modules. Notably,
since task-specific modules require few parameters, VMT-Adapter can handle an
arbitrary number of tasks with a negligible increase of trainable parameters.
We also propose VMT-Adapter-Lite, which further reduces the trainable
parameters by learning shared parameters between down- and up-projections.
Extensive experiments on four dense scene understanding tasks demonstrate the
superiority of VMT-Adapter(-Lite), achieving a 3.96%(1.34%) relative
improvement compared to single-task full fine-tuning, while utilizing merely
~1% (0.36%) trainable parameters of the pre-trained model.

大规模预训练模型已在各种计算机视觉任务中取得了显著成果。然而，与其它单一任务的适应性方法相比，在多任务适应方面的研究有限，这些方法往往表现出次优的训练和推理效率。本文首先提出了一种全能视觉多任务适配器（VMT-Adapter），其训练和推理效率与任务数量近似为 O (1)。具体而言，VMT-Adapter 通过共享多个任务的知识来增强跨任务交互，并通过独立知识提取模块保留了任务特定的知识。此外，本文还提出了 VMT-Adapter-Lite，通过学习下投影和上投影之间的共享参数来进一步减少可训练参数。对四个密集场景理解任务的大量实验证明了 VMT-Adapter (-Lite) 的优越性，相比于单一任务的全面微调，它们分别实现了 3.96%（1.34%）的相对改进，并仅利用了预训练模型的约 1%（0.36%）的可训练参数。