Semantic segmentation is a core computer vision problem, but the high costs
of data annotation have hindered its wide application. Weakly-Supervised
Semantic Segmentation (WSSS) offers a cost-efficient workaround to extensive
labeling in comparison to fully-supervised methods by using partial or
incomplete labels. Existing WSSS methods have difficulties in learning the
boundaries of objects leading to poor segmentation results. We propose a novel
and effective framework that addresses these issues by leveraging visual
foundation models inside the bounding box. Adopting a two-stage WSSS framework,
our proposed network consists of a pseudo-label generation module and a
segmentation module. The first stage leverages Segment Anything Model (SAM) to
generate high-quality pseudo-labels. To alleviate the problem of delineating
precise boundaries, we adopt SAM inside the bounding box with the help of
another pre-trained foundation model (e.g., Grounding-DINO). Furthermore, we
eliminate the necessity of using the supervision of image labels, by employing
CLIP in classification. Then in the second stage, the generated high-quality
pseudo-labels are used to train an off-the-shelf segmenter that achieves the
state-of-the-art performance on PASCAL VOC 2012 and MS COCO 2014.

使用弱监督语义分割方法解决数据标注成本高的问题，通过借鉴视觉基础模型来学习对象边界从而提高分割结果的质量。

多模式基础模型增强弱监督语义分割：一种端到端方法

Enhancing Weakly Supervised Semantic Segmentation with Multi-modal  Foundation Models: An End-to-End Approach

Weakly-Supervised Semantic Segmentation (WSSS) aims to train segmentation
models using training image data with only image-level supervision. Since
precise pixel-level annotations are not accessible, existing methods typically
focus on producing pseudo masks for training segmentation models by refining
CAM-like heatmaps. However, the produced heatmaps may only capture
discriminative image regions of target object categories or the associated
co-occurring backgrounds. To address the issues, we propose a Semantic Prompt
Learning for WSSS (SemPLeS) framework, which learns to effectively prompt the
CLIP space to enhance the semantic alignment between the segmented regions and
the target object categories. More specifically, we propose Contrastive Prompt
Learning and Class-associated Semantic Refinement to learn the prompts that
adequately describe and suppress the image backgrounds associated with each
target object category. In this way, our proposed framework is able to perform
better semantic matching between object regions and the associated text labels,
resulting in desired pseudo masks for training the segmentation model. The
proposed SemPLeS framework achieves SOTA performance on the standard WSSS
benchmarks, PASCAL VOC and MS COCO, and demonstrated interpretability with the
semantic visualization of our learned prompts. The codes will be released.

我们提出了一种名为 “SemPLeS” 的框架，通过学习有效地提示 CLIP 空间，以增强分割区域与目标对象类别之间的语义对齐，从而产生训练分割模型所需的伪掩码。在 PASCAL VOC 和 MS COCO 等标准 WSSS 基准测试中，我们所提出的 SemPLeS 框架实现了最先进的性能，并通过我们学到的提示的语义可视化展示了其解释性，代码将会发布。

SemPLeS：弱监督语义分割的语义提示学习

SemPLeS: Semantic Prompt Learning for Weakly-Supervised Semantic  Segmentation

It has been widely known that CAM (Class Activation Map) usually only
activates discriminative object regions and falsely includes lots of
object-related backgrounds. As only a fixed set of image-level object labels
are available to the WSSS (weakly supervised semantic segmentation) model, it
could be very difficult to suppress those diverse background regions consisting
of open set objects. In this paper, we propose a novel Cross Language Image
Matching (CLIMS) framework, based on the recently introduced Contrastive
Language-Image Pre-training (CLIP) model, for WSSS. The core idea of our
framework is to introduce natural language supervision to activate more
complete object regions and suppress closely-related open background regions.
In particular, we design object, background region and text label matching
losses to guide the model to excite more reasonable object regions for CAM of
each category. In addition, we design a co-occurring background suppression
loss to prevent the model from activating closely-related background regions,
with a predefined set of class-related background text descriptions. These
designs enable the proposed CLIMS to generate a more complete and compact
activation map for the target objects. Extensive experiments on PASCAL VOC2012
dataset show that our CLIMS significantly outperforms the previous
state-of-the-art methods.

本文提出了一种基于 CLIP 模型的跨语言图像匹配框架 CLIMS，通过自然语言监督以激活更合理的物体区域和抑制相关的背景区域，并使用一组预定义的类相关背景文本描述来防止模型激活相关背景区域，在 PASCAL VOC2012 数据集上进行了充分的实验，该方法明显优于之前的最先进方法。

弱监督语义分割的跨语言图像匹配

Cross Language Image Matching for Weakly Supervised Semantic Segmentation

This paper studies the problem of learning semantic segmentation from
image-level supervision only. Current popular solutions leverage object
localization maps from classifiers as supervision signals, and struggle to make
the localization maps capture more complete object content. Rather than
previous efforts that primarily focus on intra-image information, we address
the value of cross-image semantic relations for comprehensive object pattern
mining. To achieve this, two neural co-attentions are incorporated into the
classifier to complimentarily capture cross-image semantic similarities and
differences. In particular, given a pair of training images, one co-attention
enforces the classifier to recognize the common semantics from co-attentive
objects, while the other one, called contrastive co-attention, drives the
classifier to identify the unshared semantics from the rest, uncommon objects.
This helps the classifier discover more object patterns and better ground
semantics in image regions. In addition to boosting object pattern learning,
the co-attention can leverage context from other related images to improve
localization map inference, hence eventually benefiting semantic segmentation
learning. More essentially, our algorithm provides a unified framework that
handles well different WSSS settings, i.e., learning WSSS with (1) precise
image-level supervision only, (2) extra simple single-label data, and (3) extra
noisy web data. It sets new state-of-the-arts on all these settings,
demonstrating well its efficacy and generalizability. Moreover, our approach
ranked 1st place in the Weakly-Supervised Semantic Segmentation Track of
CVPR2020 Learning from Imperfect Data Challenge.

该论文研究了只从图像级别监督中学习语义分割的问题，引入了两个神经协同注意力机制来补充地捕捉跨图像的语义相似性和差异性，提高了物体模式学习和语义分割的性能，并在不同的 WSSS 设置下实现了最先进的性能。