Vision Transformers have shown great performance in single tasks such as
classification and segmentation. However, real-world problems are not isolated,
which calls for vision transformers that can perform multiple tasks
concurrently. Existing multi-task vision transformers are handcrafted and
heavily rely on human expertise. In this work, we propose a novel one-shot
neural architecture search framework, dubbed AutoTaskFormer (Automated
Multi-Task Vision TransFormer), to automate this process. AutoTaskFormer not
only identifies the weights to share across multiple tasks automatically, but
also provides thousands of well-trained vision transformers with a wide range
of parameters (e.g., number of heads and network depth) for deployment under
various resource constraints. Experiments on both small-scale (2-task
Cityscapes and 3-task NYUv2) and large-scale (16-task Taskonomy) datasets show
that AutoTaskFormer outperforms state-of-the-art handcrafted vision
transformers in multi-task learning. The entire code and models will be
open-sourced.

本研究提出了一个名为 AutoTaskFormer 的自动多任务视觉 Transformer 神经架构搜索框架，该框架能够自动识别在多个任务之间共享的权重，并在各种资源约束下提供成千上万的经过良好训练的视觉 Transformer。实验结果表明，AutoTaskFormer 在多任务学习方面优于现有的手工制作的视觉 Transformer。