The popular CLIP model displays impressive zero-shot capabilities thanks to
its seamless interaction with arbitrary text prompts. However, its lack of
spatial awareness makes it unsuitable for dense computer vision tasks, e.g.,
semantic segmentation, without an additional fine-tuning step that often uses
annotations and can potentially suppress its original open-vocabulary
properties. Meanwhile, self-supervised representation methods have demonstrated
good localization properties without human-made annotations nor explicit
supervision. In this work, we take the best of both worlds and propose a
zero-shot open-vocabulary semantic segmentation method, which does not require
any annotations. We propose to locally improve dense MaskCLIP features,
computed with a simple modification of CLIP's last pooling layer, by
integrating localization priors extracted from self-supervised features. By
doing so, we greatly improve the performance of MaskCLIP and produce smooth
outputs. Moreover, we show that the used self-supervised feature properties can
directly be learnt from CLIP features therefore allowing us to obtain the best
results with a single pass through CLIP model. Our method CLIP-DINOiser needs
only a single forward pass of CLIP and two light convolutional layers at
inference, no extra supervision nor extra memory and reaches state-of-the-art
results on challenging and fine-grained benchmarks such as COCO, Pascal
Context, Cityscapes and ADE20k. The code to reproduce our results is available
at this https URL

我们提出了一种零样本开放词汇语义分割方法，无需任何注释，通过从自监督特征中提取出的定位先验来局部改进密集的 MaskCLIP 特征，从而显著提升 MaskCLIP 的性能并产生平滑的输出。