Recently, CLIP has been applied to pixel-level zero-shot learning tasks via a
two-stage scheme. The general idea is to first generate class-agnostic region
proposals and then feed the cropped proposal regions to CLIP to utilize its
image-level zero-shot classification capability. While effective, such a scheme
requires two image encoders, one for proposal generation and one for CLIP,
leading to a complicated pipeline and high computational cost. In this work, we
pursue a simpler-and-efficient one-stage solution that directly extends CLIP's
zero-shot prediction capability from image to pixel level. Our investigation
starts with a straightforward extension as our baseline that generates semantic
masks by comparing the similarity between text and patch embeddings extracted
from CLIP. However, such a paradigm could heavily overfit the seen classes and
fail to generalize to unseen classes. To handle this issue, we propose three
simple-but-effective designs and figure out that they can significantly retain
the inherent zero-shot capacity of CLIP and improve pixel-level generalization
ability. Incorporating those modifications leads to an efficient zero-shot
semantic segmentation system called ZegCLIP. Through extensive experiments on
three public benchmarks, ZegCLIP demonstrates superior performance,
outperforming the state-of-the-art methods by a large margin under both
"inductive" and "transductive" zero-shot settings. In addition, compared with
the two-stage method, our one-stage ZegCLIP achieves a speedup of about 5 times
faster during inference. We release the code at
this https URL

本文提出了一种基于 CLIP 的零样本语义分割方法 ZegCLIP，将其从图像级别扩展到像素级别，通过三种简单而有效的设计处理过拟合问题，具有更好的泛化能力和速度优势。