The Segment Anything Model (SAM), with its remarkable zero-shot capability,
has been proven to be a powerful foundation model for image segmentation tasks,
which is an important task in computer vision. However, the transfer of its
rich semantic information to multiple different downstream tasks remains
unexplored. In this paper, we propose the Task-Aware Low-Rank Adaptation
(TA-LoRA) method, which enables SAM to work as a foundation model for
multi-task learning. Specifically, TA-LoRA injects an update parameter tensor
into each layer of the encoder in SAM and leverages a low-rank tensor
decomposition method to incorporate both task-shared and task-specific
information. Furthermore, we introduce modified SAM (mSAM) for multi-task
learning where we remove the prompt encoder of SAM and use task-specific no
mask embeddings and mask decoder for each task. Extensive experiments conducted
on benchmark datasets substantiate the efficacy of TA-LoRA in enhancing the
performance of mSAM across multiple downstream tasks.

本文介绍了一种新的方法 TA-LoRA，将 Segment Anything Model (SAM) 用作多任务学习的基础模型，在多个下游任务中显著提高了 mSAM 的性能。