In-context segmentation has drawn more attention with the introduction of
vision foundation models. Most existing approaches adopt metric learning or
masked image modeling to build the correlation between visual prompts and input
image queries. In this work, we explore this problem from a new perspective,
using one representative generation model, the latent diffusion model (LDM). We
observe a task gap between generation and segmentation in diffusion models, but
LDM is still an effective minimalist for in-context segmentation. In
particular, we propose two meta-architectures and correspondingly design
several output alignment and optimization strategies. We have conducted
comprehensive ablation studies and empirically found that the segmentation
quality counts on output alignment and in-context instructions. Moreover, we
build a new and fair in-context segmentation benchmark that includes both image
and video datasets. Experiments validate the efficiency of our approach,
demonstrating comparable or even stronger results than previous specialist
models or visual foundation models. Our study shows that LDMs can also achieve
good enough results for challenging in-context segmentation tasks.

通过使用代表性的生成模型 —— 潜在扩散模型（LDM），我们从新的角度探索了上下文分割问题，提出了两种元架构和相应的输出对齐和优化策略。我们通过全面的消融研究发现，分割质量取决于输出对齐和上下文指令。实验证明，我们的方法在挑战性的上下文分割任务中取得了与专业模型或视觉基础模型相当甚至更强的结果，证明了潜在扩散模型也可以达到足够好的效果。

通过潜在扩散模型探索上下文分割

Explore In-Context Segmentation via Latent Diffusion Models

In-context segmentation aims at segmenting novel images using a few labeled
example images, termed as "in-context examples", exploring content similarities
between examples and the target. The resulting models can be generalized
seamlessly to novel segmentation tasks, significantly reducing the labeling and
training costs compared with conventional pipelines. However, in-context
segmentation is more challenging than classic ones due to its meta-learning
nature, requiring the model to learn segmentation rules conditioned on a few
samples, not just the segmentation. Unlike previous work with ad-hoc or
non-end-to-end designs, we propose SEGIC, an end-to-end segment-in-context
framework built upon a single vision foundation model (VFM). In particular,
SEGIC leverages the emergent correspondence within VFM to capture dense
relationships between target images and in-context samples. As such,
information from in-context samples is then extracted into three types of
instructions, i.e. geometric, visual, and meta instructions, serving as
explicit conditions for the final mask prediction. SEGIC is a straightforward
yet effective approach that yields state-of-the-art performance on one-shot
segmentation benchmarks. Notably, SEGIC can be easily generalized to diverse
tasks, including video object segmentation and open-vocabulary segmentation.
Code will be available at https://github.com/MengLcool/SEGIC.

利用上下文分割框架（SEGIC）的元学习和端到端设计，可以实现在只有少数示例图像的情况下对新图像进行分割，并显著降低标记和训练成本，同时在一次性分割基准测试中取得最先进的性能。