To bridge the gap between supervised semantic segmentation and real-world
applications that acquires one model to recognize arbitrary new concepts,
recent zero-shot segmentation attracts a lot of attention by exploring the
relationships between unseen and seen object categories, yet requiring large
amounts of densely-annotated data with diverse base classes. In this paper, we
propose a new open-world semantic segmentation pipeline that makes the first
attempt to learn to segment semantic objects of various open-world categories
without any efforts on dense annotations, by purely exploiting the
image-caption data that naturally exist on the Internet. Our method,
Vision-language-driven Semantic Segmentation (ViL-Seg), employs an image and a
text encoder to generate visual and text embeddings for the image-caption data,
with two core components that endow its segmentation ability: First, the image
encoder is jointly trained with a vision-based contrasting and a cross-modal
contrasting, which encourage the visual embeddings to preserve both
fine-grained semantics and high-level category information that are crucial for
the segmentation task. Furthermore, an online clustering head is devised over
the image encoder, which allows to dynamically segment the visual embeddings
into distinct semantic groups such that they can be classified by comparing
with various text embeddings to complete our segmentation pipeline. Experiments
show that without using any data with dense annotations, our method can
directly segment objects of arbitrary categories, outperforming zero-shot
segmentation methods that require data labeling on three benchmark datasets.

本文提出了一种基于图像文本交互的语义分割模型 ViL-Seg，通过无需密集标注的方式，利用网络上自然存在的图像和文本数据，学习到能够直接分割任意开放世界类别对象的能力，实验结果在三个基准数据集上优于需要数据标注的零样本分割方法。

通过对比和聚类视觉语言嵌入进行开放世界语义分割

Open-world Semantic Segmentation via Contrasting and Clustering  Vision-Language Embedding

Object referring has important applications, especially for human-machine
interaction. While having received great attention, the task is mainly attacked
with written language (text) as input rather than spoken language (speech),
which is more natural. This paper investigates Object Referring with Spoken
Language (ORSpoken) by presenting two datasets and one novel approach. Objects
are annotated with their locations in images, text descriptions and speech
descriptions. This makes the datasets ideal for multi-modality learning. The
approach is developed by carefully taking down ORSpoken problem into three
sub-problems and introducing task-specific vision-language interactions at the
corresponding levels. Experiments show that our method outperforms competing
methods consistently and significantly. The approach is also evaluated in the
presence of audio noise, showing the efficacy of the proposed vision-language
interaction methods in counteracting background noise.

本文探讨了用口语作为输入的物体指称（ORSpoken），通过介绍两个数据集和一种新的方法来为多模式学习提供了理想的数据集，并在相应的层次引入任务特定的视觉语言交互，实验表明我们的方法在减轻背景噪声方面具有很好的效果。