The open-vocabulary image segmentation task involves partitioning images into
semantically meaningful segments and classifying them with flexible
text-defined categories. The recent vision-based foundation models such as the
Segment Anything Model (SAM) have shown superior performance in generating
class-agnostic image segments. The main challenge in open-vocabulary image
segmentation now lies in accurately classifying these segments into
text-defined categories. In this paper, we introduce the Universal Segment
Embedding (USE) framework to address this challenge. This framework is
comprised of two key components: 1) a data pipeline designed to efficiently
curate a large amount of segment-text pairs at various granularities, and 2) a
universal segment embedding model that enables precise segment classification
into a vast range of text-defined categories. The USE model can not only help
open-vocabulary image segmentation but also facilitate other downstream tasks
(e.g., querying and ranking). Through comprehensive experimental studies on
semantic segmentation and part segmentation benchmarks, we demonstrate that the
USE framework outperforms state-of-the-art open-vocabulary segmentation
methods.

我们介绍了通用段落嵌入（USE）框架，该框架通过数据管道和通用段落嵌入模型，能够准确地将图像划分为各种文本定义的类别，并在开放词汇图像分割和其他下游任务中取得更好的性能。