Segment Anything Model (SAM) has attracted widespread attention for its
superior interactive segmentation capabilities with visual prompts while
lacking further exploration of text prompts. In this paper, we empirically
investigate what text prompt encoders (e.g., CLIP or LLM) are good for adapting
SAM for referring expression segmentation and introduce the Early
Vision-language Fusion-based SAM (EVF-SAM). EVF-SAM is a simple yet effective
referring segmentation method which exploits multimodal prompts (i.e., image
and text) and comprises a pre-trained vision-language model to generate
referring prompts and a SAM model for segmentation. Surprisingly, we observe
that: (1) multimodal prompts and (2) vision-language models with early fusion
(e.g., BEIT-3) are beneficial for prompting SAM for accurate referring
segmentation. Our experiments show that the proposed EVF-SAM based on BEIT-3
can obtain state-of-the-art performance on RefCOCO/+/g for referring expression
segmentation and demonstrate the superiority of prompting SAM with early
vision-language fusion. In addition, the proposed EVF-SAM with 1.32B parameters
achieves remarkably higher performance while reducing nearly 82% of parameters
compared to previous SAM methods based on large multimodal models.

利用多模态提示和视觉 - 语言模型，本文介绍了一种简单而有效的基于早期视觉 - 语言融合的提及分割方法，EVF-SAM，它能以较少的参数获得比以往的大型多模态模型更好的性能，用于提高 Segment Anything Model (SAM) 在提及分割任务上的表现。