Instance segmentation datasets play a crucial role in training accurate and
robust computer vision models. However, obtaining accurate mask annotations to
produce high-quality segmentation datasets is a costly and labor-intensive
process. In this work, we show how this issue can be mitigated by starting with
small annotated instance segmentation datasets and augmenting them to
effectively obtain a sizeable annotated dataset. We achieve that by creating
variations of the available annotated object instances in a way that preserves
the provided mask annotations, thereby resulting in new image-mask pairs to be
added to the set of annotated images. Specifically, we generate new images
using a diffusion-based inpainting model to fill out the masked area with a
desired object class by guiding the diffusion through the object outline. We
show that the object outline provides a simple, but also reliable and
convenient training-free guidance signal for the underlying inpainting model
that is often sufficient to fill out the mask with an object of the correct
class without further text guidance and preserve the correspondence between
generated images and the mask annotations with high precision. Our experimental
results reveal that our method successfully generates realistic variations of
object instances, preserving their shape characteristics while introducing
diversity within the augmented area. We also show that the proposed method can
naturally be combined with text guidance and other image augmentation
techniques.

使用扩充数据集的方法，通过创建基于注释目标实例的多样化图像和标记对来对实例分割数据集进行增强，同时保留原有标记的一致性。实验结果显示该方法成功生成了实际的目标实例变体，并在扩增区域内引入了多样性。同时，该方法可以与文本指导和其他图像增强技术自然地结合。

基于轮廓引导的扩散模型目标修复

Outline-Guided Object Inpainting with Diffusion Models

Existing open-vocabulary image segmentation methods require a fine-tuning
step on mask annotations and/or image-text datasets. Mask labels are
labor-intensive, which limits the number of categories in segmentation
datasets. As a result, the open-vocabulary capacity of pre-trained VLMs is
severely reduced after fine-tuning. However, without fine-tuning, VLMs trained
under weak image-text supervision tend to make suboptimal mask predictions when
there are text queries referring to non-existing concepts in the image. To
alleviate these issues, we introduce a novel recurrent framework that
progressively filters out irrelevant texts and enhances mask quality without
training efforts. The recurrent unit is a two-stage segmenter built upon a VLM
with frozen weights. Thus, our model retains the VLM's broad vocabulary space
and strengthens its segmentation capability. Experimental results show that our
method outperforms not only the training-free counterparts, but also those
fine-tuned with millions of additional data samples, and sets new
state-of-the-art records for both zero-shot semantic and referring image
segmentation tasks. Specifically, we improve the current record by 28.8, 16.0,
and 6.9 mIoU on Pascal VOC, COCO Object, and Pascal Context.

通过引入一个创新的递归框架，我们的研究表明，在不进行训练的情况下，我们的模型能够优于那些经过百万级附加样本微调的方法，为零样本语义和参考图像分割任务的最新记录设定了新的技术水平。

CLIP 作为 RNN：无需培训即可分割无限的视觉概念

CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor

Open-vocabulary instance segmentation aims at segmenting novel classes
without mask annotations. It is an important step toward reducing laborious
human supervision. Most existing works first pretrain a model on captioned
images covering many novel classes and then finetune it on limited base classes
with mask annotations. However, the high-level textual information learned from
caption pretraining alone cannot effectively encode the details required for
pixel-wise segmentation. To address this, we propose a cross-modal
pseudo-labeling framework, which generates training pseudo masks by aligning
word semantics in captions with visual features of object masks in images.
Thus, our framework is capable of labeling novel classes in captions via their
word semantics to self-train a student model. To account for noises in pseudo
masks, we design a robust student model that selectively distills mask
knowledge by estimating the mask noise levels, hence mitigating the adverse
impact of noisy pseudo masks. By extensive experiments, we show the
effectiveness of our framework, where we significantly improve mAP score by
4.5% on MS-COCO and 5.1% on the large-scale Open Images & Conceptual Captions
datasets compared to the state-of-the-art.

本文提出一种跨模态伪标签（cross-modal pseudo-labeling）框架，用于面向开放词汇的实例分割，通过与对象掩膜的视觉特征对齐，实现对字词语义中的新类别进行标记，从而自我训练出一个学生模型，缓解了伪掩膜中存在的噪声干扰所带来的不良影响，相比现有工作，在 MS-COCO 和 Open Images & Conceptual Captions 数据集上分别实现了 4.5% 和 5.1% 的 mAP 得分提高。