Existing Generalized Zero-shot Semantic Segmentation (GZLSS) methods apply
either finetuning the CLIP paradigm or formulating it as a mask classification
task, benefiting from the Vision-Language Models (VLMs). However, the
fine-tuning methods are restricted with fixed backbone models which are not
flexible for segmentation, and mask classification methods heavily rely on
additional explicit mask proposers. Meanwhile, prevalent methods utilize only
seen categories which is a great waste, i.e., neglecting the area exists but
not annotated. To this end, we propose CLIPTeacher, a new learning framework
that can be applied to various per-pixel classification segmentation models
without introducing any explicit mask proposer or changing the structure of
CLIP, and utilize both seen and ignoring areas. Specifically, CLIPTeacher
consists of two key modules: Global Learning Module (GLM) and Pixel Learning
Module (PLM). Specifically, GLM aligns the dense features from an image encoder
with the CLS token, i.e., the only token trained in CLIP, which is a simple but
effective way to probe global information from the CLIP models. In contrast,
PLM only leverages dense tokens from CLIP to produce high-level pseudo
annotations for ignoring areas without introducing any extra mask proposer.
Meanwhile, PLM can fully take advantage of the whole image based on the pseudo
annotations. Experimental results on three benchmark datasets: PASCAL VOC 2012,
COCO-Stuff 164k, and PASCAL Context show large performance gains, i.e., 2.2%,
1.3%, and 8.8%

提出了一种名为 CLIPTeacher 的新学习框架，用于各种基于像素分类的分割模型，不引入任何显式的掩码建议者或更改 CLIP 的结构，并利用已知区域和忽略区域来取得较大的性能提升。