It has been widely known that CAM (Class Activation Map) usually only
activates discriminative object regions and falsely includes lots of
object-related backgrounds. As only a fixed set of image-level object labels
are available to the WSSS (weakly supervised semantic segmentation) model, it
could be very difficult to suppress those diverse background regions consisting
of open set objects. In this paper, we propose a novel Cross Language Image
Matching (CLIMS) framework, based on the recently introduced Contrastive
Language-Image Pre-training (CLIP) model, for WSSS. The core idea of our
framework is to introduce natural language supervision to activate more
complete object regions and suppress closely-related open background regions.
In particular, we design object, background region and text label matching
losses to guide the model to excite more reasonable object regions for CAM of
each category. In addition, we design a co-occurring background suppression
loss to prevent the model from activating closely-related background regions,
with a predefined set of class-related background text descriptions. These
designs enable the proposed CLIMS to generate a more complete and compact
activation map for the target objects. Extensive experiments on PASCAL VOC2012
dataset show that our CLIMS significantly outperforms the previous
state-of-the-art methods.

本文提出了一种基于 CLIP 模型的跨语言图像匹配框架 CLIMS，通过自然语言监督以激活更合理的物体区域和抑制相关的背景区域，并使用一组预定义的类相关背景文本描述来防止模型激活相关背景区域，在 PASCAL VOC2012 数据集上进行了充分的实验，该方法明显优于之前的最先进方法。