We propose an end-to-end Multitask Learning Transformer framework, named
MulT, to simultaneously learn multiple high-level vision tasks, including depth
estimation, semantic segmentation, reshading, surface normal estimation, 2D
keypoint detection, and edge detection. Based on the Swin transformer model,
our framework encodes the input image into a shared representation and makes
predictions for each vision task using task-specific transformer-based decoder
heads. At the heart of our approach is a shared attention mechanism modeling
the dependencies across the tasks. We evaluate our model on several multitask
benchmarks, showing that our MulT framework outperforms both the state-of-the
art multitask convolutional neural network models and all the respective single
task transformer models. Our experiments further highlight the benefits of
sharing attention across all the tasks, and demonstrate that our MulT model is
robust and generalizes well to new domains. Our project website is at
this https URL

我们开发了一种称为 MulT 的端到端多任务学习 Transformer 框架，用于同时学习多个高级视觉任务，包括深度估计、语义分割、重着色、表面法线估计、2D 关键点检测和边缘检测。我们的框架基于 Swin transformer 模型，利用任务特定的基于 Transformer 的解码器头为每个视觉任务进行预测，并采用共享的注意机制模型来建模跨任务的依赖关系。我们在多个多任务基准测试中评估了我们的模型，显示出我们的 MulT 框架优于最先进的多任务卷积神经网络模型和所有各自的单任务 Transformer 模型。我们的实验进一步突出了共享所有任务的注意力的优点，并证明我们的 MulT 模型具有鲁棒性和很好的泛化性能。