We introduce the first multitasking vision transformer adapters that learn
generalizable task affinities which can be applied to novel tasks and domains.
Integrated into an off-the-shelf vision transformer backbone, our adapters can
simultaneously solve multiple dense vision tasks in a parameter-efficient
manner, unlike existing multitasking transformers that are parametrically
expensive. In contrast to concurrent methods, we do not require retraining or
fine-tuning whenever a new task or domain is added. We introduce a task-adapted
attention mechanism within our adapter framework that combines gradient-based
task similarities with attention-based ones. The learned task affinities
generalize to the following settings: zero-shot task transfer, unsupervised
domain adaptation, and generalization without fine-tuning to novel domains. We
demonstrate that our approach outperforms not only the existing convolutional
neural network-based multitasking methods but also the vision transformer-based
ones. Our project page is at https://ivrl.github.io/VTAGML.

我们引入了第一款多任务视觉变换器适配器，它能学习到适用于新任务和领域的通用任务关联性。我们的适配器融合在现成的视觉变换器骨干网络中，能够同时解决多个密集视觉任务，而不像现有的多任务变换器那样需要大量的参数。与同时方法相比，我们无需在添加新任务或领域时重新训练或微调。我们在适配器框架中引入了一种任务适应的注意机制，将基于梯度的任务相似度与基于注意力的相似度相结合。所学习的任务关联性可推广到以下情况：零样本任务迁移、无监督领域适应以及不需要对新领域进行微调的泛化。我们证明了我们的方法不仅优于现有的基于卷积神经网络的多任务方法，还优于基于视觉变换器的方法。