This paper studies open-vocabulary segmentation (OVS) through calibrating
in-vocabulary and domain-biased embedding space with generalized contextual
prior of CLIP. As the core of open-vocabulary understanding, alignment of
visual content with the semantics of unbounded text has become the bottleneck
of this field. To address this challenge, recent works propose to utilize CLIP
as an additional classifier and aggregate model predictions with CLIP
classification results. Despite their remarkable progress, performance of OVS
methods in relevant scenarios is still unsatisfactory compared with supervised
counterparts. We attribute this to the in-vocabulary embedding and
domain-biased CLIP prediction. To this end, we present a Semantic-assisted
CAlibration Network (SCAN). In SCAN, we incorporate generalized semantic prior
of CLIP into proposal embedding to avoid collapsing on known categories.
Besides, a contextual shift strategy is applied to mitigate the lack of global
context and unnatural background noise. With above designs, SCAN achieves
state-of-the-art performance on all popular open-vocabulary segmentation
benchmarks. Furthermore, we also focus on the problem of existing evaluation
system that ignores semantic duplication across categories, and propose a new
metric called Semantic-Guided IoU (SG-IoU).

本文通过使用 CLIP 的广义上下文先验，通过校准在词汇内和领域偏置的嵌入空间，研究开放词汇分割（OVS）。为了解决这一挑战，我们提出了一个称为 Semantic-assisted CAlibration Network（SCAN）的方法，并引入了广义语义先验以避免在已知类别上崩溃。此外，我们还提出了一种名为 Semantic-Guided IoU（SG-IoU）的新度量标准，以解决现有评估系统忽略分类间语义重复的问题。