The variety of objects in the real world is nearly unlimited and is thus
impossible to capture using models trained on a fixed set of categories. As a
result, in recent years, open-vocabulary methods have attracted the interest of
the community. This paper proposes a new method for zero-shot open-vocabulary
segmentation. Prior work largely relies on contrastive training using
image-text pairs, leveraging grouping mechanisms to learn image features that
are both aligned with language and well-localised. This however can introduce
ambiguity as the visual appearance of images with similar captions often
varies. Instead, we leverage the generative properties of large-scale
text-to-image diffusion models to sample a set of support images for a given
textual category. This provides a distribution of appearances for a given text
circumventing the ambiguity problem. We further propose a mechanism that
considers the contextual background of the sampled images to better localise
objects and segment the background directly. We show that our method can be
used to ground several existing pre-trained self-supervised feature extractors
in natural language and provide explainable predictions by mapping back to
regions in the support set. Our proposal is training-free, relying on
pre-trained components only, yet, shows strong performance on a range of
open-vocabulary segmentation benchmarks, obtaining a lead of more than 10% on
the Pascal VOC benchmark.

本文提出了一种基于大规模文本到图像扩散模型的零样本开放词库分割方法，通过考虑采样的图像的背景来更好地定位对象并直接分割背景，并证明该方法可以在多个基准测试中表现出色，在 Pascal VOC 基准测试上领先了超过 10%。

零样本开放词汇分割的扩散模型

Diffusion Models for Zero-Shot Open-Vocabulary Segmentation

Methods that move towards less supervised scenarios are key for image
segmentation, as dense labels demand significant human intervention. Generally,
the annotation burden is mitigated by labeling datasets with weaker forms of
supervision, e.g. image-level labels or bounding boxes. Another option are
semi-supervised settings, that commonly leverage a few strong annotations and a
huge number of unlabeled/weakly-labeled data. In this paper, we revisit
semi-supervised segmentation schemes and narrow down significantly the
annotation budget (in terms of total labeling time of the training set)
compared to previous approaches. With a very simple pipeline, we demonstrate
that at low annotation budgets, semi-supervised methods outperform by a wide
margin weakly-supervised ones for both semantic and instance segmentation. Our
approach also outperforms previous semi-supervised works at a much reduced
labeling cost. We present results for the Pascal VOC benchmark and unify weakly
and semi-supervised approaches by considering the total annotation budget, thus
allowing a fairer comparison between methods.

本文研究了半监督分割方法，与之前的方法相比，本文的方法在标注预算方面有了重大突破，在使用低标注预算时，我们的方法在语义与实例分割方面表现优于弱监督方法，并在较小的标注成本下优于以往的半监督方法。同时，我们通过考虑总的标注预算，将弱监督和半监督方法结合在一起，从而更公正地比较各种方法。我们在 Pascal VOC 基准测试上展示了结果。

预算感知半监督语义与实例分割

Budget-aware Semi-Supervised Semantic and Instance Segmentation

We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.

该研究通过在深度网络中嵌入结构化的 CRF，将包含目标检测和超像素等等高级潜在因素的方法应用于语义分割任务中，实现了 PASCAL VOC 基准上最佳分割性能。