The integration of diverse visual prompts like clicks, scribbles, and boxes
in interactive image segmentation could significantly facilitate user
interaction as well as improve interaction efficiency. Most existing studies
focus on a single type of visual prompt by simply concatenating prompts and
images as input for segmentation prediction, which suffers from low-efficiency
prompt representation and weak interaction issues. This paper proposes a simple
yet effective Visual Prompt Unified Transformer (VPUFormer), which introduces a
concise unified prompt representation with deeper interaction to boost the
segmentation performance. Specifically, we design a Prompt-unified Encoder
(PuE) by using Gaussian mapping to generate a unified one-dimensional vector
for click, box, and scribble prompts, which well captures users' intentions as
well as provides a denser representation of user prompts. In addition, we
present a Prompt-to-Pixel Contrastive Loss (P2CL) that leverages user feedback
to gradually refine candidate semantic features, aiming to bring image semantic
features closer to the features that are similar to the user prompt, while
pushing away those image semantic features that are dissimilar to the user
prompt, thereby correcting results that deviate from expectations. On this
basis, our approach injects prompt representations as queries into Dual-cross
Merging Attention (DMA) blocks to perform a deeper interaction between image
and query inputs. A comprehensive variety of experiments on seven challenging
datasets demonstrates that the proposed VPUFormer with PuE, DMA, and P2CL
achieves consistent improvements, yielding state-of-the-art segmentation
performance. Our code will be made publicly available at
this https URL

该论文提出了一个简单但高效的视觉提示统一变压器，利用更深入的交互来提高图像分割性能，并使用高斯映射生成一个统一的一维向量来捕获用户的意图，并利用用户反馈逐渐改进候选语义特征，从而逐步提高图像分割性能。