The realm of computer vision has witnessed a paradigm shift with the advent
of foundational models, mirroring the transformative influence of large
language models in the domain of natural language processing. This paper delves
into the exploration of open-world segmentation, presenting a novel approach
called Image Prompt Segmentation (IPSeg) that harnesses the power of vision
foundational models. At the heart of IPSeg lies the principle of a
training-free paradigm, which capitalizes on image prompting techniques. IPSeg
utilizes a single image containing a subjective visual concept as a flexible
prompt to query vision foundation models like DINOv2 and Stable Diffusion. Our
approach extracts robust features for the prompt image and input image, then
matches the input representations to the prompt representations via a novel
feature interaction module to generate point prompts highlighting target
objects in the input image. The generated point prompts are further utilized to
guide the Segment Anything Model to segment the target object in the input
image. The proposed method stands out by eliminating the need for exhaustive
training sessions, thereby offering a more efficient and scalable solution.
Experiments on COCO, PASCAL VOC, and other datasets demonstrate IPSeg's
efficacy for flexible open-world segmentation using intuitive image prompts.
This work pioneers tapping foundation models for open-world understanding
through visual concepts conveyed in images.

通过使用图像概念传达的视觉概念，本研究首次探索了使用基础模型进行开放世界理解的方法，提出了一种名为图像提示分割（IPSeg）的新方法，该方法利用了图像提示技术，通过提取强大的特征，并通过新颖的特征交互模块将输入图像的表示与提示图像的表示进行匹配，以生成突出显示输入图像中目标对象的点提示，并进一步利用生成的点提示来指导 Segment Anything Model 对输入图像中的目标对象进行分割，从而消除了繁琐的训练过程，提供更高效和可扩展的解决方案。