Weakly supervised semantic segmentation (WSSS) with image-level labels is a
challenging task. Mainstream approaches follow a multi-stage framework and
suffer from high training costs. In this paper, we explore the potential of
Contrastive Language-Image Pre-training models (CLIP) to localize different
categories with only image-level labels and without further training. To
efficiently generate high-quality segmentation masks from CLIP, we propose a
novel WSSS framework called CLIP-ES. Our framework improves all three stages of
WSSS with special designs for CLIP: 1) We introduce the softmax function into
GradCAM and exploit the zero-shot ability of CLIP to suppress the confusion
caused by non-target classes and backgrounds. Meanwhile, to take full advantage
of CLIP, we re-explore text inputs under the WSSS setting and customize two
text-driven strategies: sharpness-based prompt selection and synonym fusion. 2)
To simplify the stage of CAM refinement, we propose a real-time class-aware
attention-based affinity (CAA) module based on the inherent multi-head
self-attention (MHSA) in CLIP-ViTs. 3) When training the final segmentation
model with the masks generated by CLIP, we introduced a confidence-guided loss
(CGL) focus on confident regions. Our CLIP-ES achieves SOTA performance on
Pascal VOC 2012 and MS COCO 2014 while only taking 10% time of previous methods
for the pseudo mask generation. Code is available at
this https URL.

利用 Contrastive Language-Image Pre-training (简称 CLIP) 模型进行图像级标注的弱监督语义分割 (WSSS) 是一项具有挑战性的任务。本文提出了一种 CLIP-ES 框架，结合了 softmax 函数、基于文本的驱动策略和一种实时的基于注意力机制的亲和度 (CAA) 模块，以及 CGL 损失，来提高 WSSS 的效率和性能。