The realm of computer vision has witnessed a paradigm shift with the advent
of foundational models, mirroring the transformative influence of large
language models in the domain of natural language processing. This paper delves
into the exploration of open-world segmentation, presenting a novel approach
called Image Prompt Segmentation (IPSeg) that harnesses the power of vision
foundational models. At the heart of IPSeg lies the principle of a
training-free paradigm, which capitalizes on image prompting techniques. IPSeg
utilizes a single image containing a subjective visual concept as a flexible
prompt to query vision foundation models like DINOv2 and Stable Diffusion. Our
approach extracts robust features for the prompt image and input image, then
matches the input representations to the prompt representations via a novel
feature interaction module to generate point prompts highlighting target
objects in the input image. The generated point prompts are further utilized to
guide the Segment Anything Model to segment the target object in the input
image. The proposed method stands out by eliminating the need for exhaustive
training sessions, thereby offering a more efficient and scalable solution.
Experiments on COCO, PASCAL VOC, and other datasets demonstrate IPSeg's
efficacy for flexible open-world segmentation using intuitive image prompts.
This work pioneers tapping foundation models for open-world understanding
through visual concepts conveyed in images.

通过使用图像概念传达的视觉概念，本研究首次探索了使用基础模型进行开放世界理解的方法，提出了一种名为图像提示分割（IPSeg）的新方法，该方法利用了图像提示技术，通过提取强大的特征，并通过新颖的特征交互模块将输入图像的表示与提示图像的表示进行匹配，以生成突出显示输入图像中目标对象的点提示，并进一步利用生成的点提示来指导 Segment Anything Model 对输入图像中的目标对象进行分割，从而消除了繁琐的训练过程，提供更高效和可扩展的解决方案。

通过图像提示构建训练无关的开放世界分割基础模型

Towards Training-free Open-world Segmentation via Image Prompting  Foundation Models

To bridge the gap between supervised semantic segmentation and real-world
applications that acquires one model to recognize arbitrary new concepts,
recent zero-shot segmentation attracts a lot of attention by exploring the
relationships between unseen and seen object categories, yet requiring large
amounts of densely-annotated data with diverse base classes. In this paper, we
propose a new open-world semantic segmentation pipeline that makes the first
attempt to learn to segment semantic objects of various open-world categories
without any efforts on dense annotations, by purely exploiting the
image-caption data that naturally exist on the Internet. Our method,
Vision-language-driven Semantic Segmentation (ViL-Seg), employs an image and a
text encoder to generate visual and text embeddings for the image-caption data,
with two core components that endow its segmentation ability: First, the image
encoder is jointly trained with a vision-based contrasting and a cross-modal
contrasting, which encourage the visual embeddings to preserve both
fine-grained semantics and high-level category information that are crucial for
the segmentation task. Furthermore, an online clustering head is devised over
the image encoder, which allows to dynamically segment the visual embeddings
into distinct semantic groups such that they can be classified by comparing
with various text embeddings to complete our segmentation pipeline. Experiments
show that without using any data with dense annotations, our method can
directly segment objects of arbitrary categories, outperforming zero-shot
segmentation methods that require data labeling on three benchmark datasets.

本文提出了一种基于图像文本交互的语义分割模型 ViL-Seg，通过无需密集标注的方式，利用网络上自然存在的图像和文本数据，学习到能够直接分割任意开放世界类别对象的能力，实验结果在三个基准数据集上优于需要数据标注的零样本分割方法。