Recently, pre-trained vision-language models have been increasingly used to
tackle the challenging zero-shot segmentation task. Typical solutions follow
the paradigm of first generating mask proposals and then adopting CLIP to
classify them. To maintain the CLIP's zero-shot transferability, previous
practices favour to freeze CLIP during training. However, in the paper, we
reveal that CLIP is insensitive to different mask proposals and tends to
produce similar predictions for various mask proposals of the same image. This
insensitivity results in numerous false positives when classifying mask
proposals. This issue mainly relates to the fact that CLIP is trained with
image-level supervision. To alleviate this issue, we propose a simple yet
effective method, named Mask-aware Fine-tuning (MAFT). Specifically,
Image-Proposals CLIP Encoder (IP-CLIP Encoder) is proposed to handle arbitrary
numbers of image and mask proposals simultaneously. Then, mask-aware loss and
self-distillation loss are designed to fine-tune IP-CLIP Encoder, ensuring CLIP
is responsive to different mask proposals while not sacrificing
transferability. In this way, mask-aware representations can be easily learned
to make the true positives stand out. Notably, our solution can seamlessly plug
into most existing methods without introducing any new parameters during the
fine-tuning process. We conduct extensive experiments on the popular zero-shot
benchmarks. With MAFT, the performance of the state-of-the-art methods is
promoted by a large margin: 50.4% (+ 8.2%) on COCO, 81.8% (+ 3.2%) on
Pascal-VOC, and 8.7% (+4.3%) on ADE20K in terms of mIoU for unseen classes. The
code is available at this https URL

提出了一种名为 Mask-aware Fine-tuning (MAFT) 的简单而有效方法，通过使用 Image-Proposals CLIP Encoder (IP-CLIP Encoder) 和相应的损失函数来改善预训练模型 CLIP 对不同掩码提议的敏感性，以解决零样本分割任务中存在的虚假阳性问题。在常见的零样本基准测试中，该方法在未见过的类别上大幅提升了最先进方法的性能。

学习面具感知 CLIP 表示以进行零样本分割

Learning Mask-aware CLIP Representations for Zero-Shot Segmentation

This paper presents a new framework for open-vocabulary semantic segmentation
with the pre-trained vision-language model, named Side Adapter Network (SAN).
Our approach models the semantic segmentation task as a region recognition
problem. A side network is attached to a frozen CLIP model with two branches:
one for predicting mask proposals, and the other for predicting attention bias
which is applied in the CLIP model to recognize the class of masks. This
decoupled design has the benefit CLIP in recognizing the class of mask
proposals. Since the attached side network can reuse CLIP features, it can be
very light. In addition, the entire network can be trained end-to-end, allowing
the side network to be adapted to the frozen CLIP model, which makes the
predicted mask proposals CLIP-aware. Our approach is fast, accurate, and only
adds a few additional trainable parameters. We evaluate our approach on
multiple semantic segmentation benchmarks. Our method significantly outperforms
other counterparts, with up to 18 times fewer trainable parameters and 19 times
faster inference speed. We hope our approach will serve as a solid baseline and
help ease future research in open-vocabulary semantic segmentation. The code
will be available at this https URL.

本文介绍了一种利用预训练的视觉 - 语言模型（名为 Side Adapter Network）实现开放词汇语义分割的新框架，其中包括对固定 CLIP 模型附加侧网络以识别类别和预测掩模建议等过程，最终实现快速而精确的语义分割，并在多个基准测试集上超越了其他竞争者，还拥有较少的可训练参数和较快的推断速度。

面向开放词汇语义分割的旁路适配器网络

Side Adapter Network for Open-Vocabulary Semantic Segmentation

Open-world instance segmentation is the task of grouping pixels into object
instances without any pre-determined taxonomy. This is challenging, as
state-of-the-art methods rely on explicit class semantics obtained from large
labeled datasets, and out-of-domain evaluation performance drops significantly.
Here we propose a novel approach for mask proposals, Generic Grouping Networks
(GGNs), constructed without semantic supervision. Our approach combines a local
measure of pixel affinity with instance-level mask supervision, producing a
training regimen designed to make the model as generic as the data diversity
allows. We introduce a method for predicting Pairwise Affinities (PA), a
learned local relationship between pairs of pixels. PA generalizes very well to
unseen categories. From PA we construct a large set of pseudo-ground-truth
instance masks; combined with human-annotated instance masks we train GGNs and
significantly outperform the SOTA on open-world instance segmentation on
various benchmarks including COCO, LVIS, ADE20K, and UVO. Code is available on
project website: this https URL

本文提出一种能够在没有先验标签的情况下实现像素实例分割的新方法：Generic Grouping Networks，在预测配对亲和性（PA）的基础上构建伪基础实例掩码，并与真实标注掩码一起训练模型，从而在多个基准数据集上，相较于现有最优方法，实现了更好的开放域实例分割性能。