Weakly supervised semantic segmentation (WSSS), a fundamental computer vision
task, which aims to segment out the object within only class-level labels. The
traditional methods adopt the CNN-based network and utilize the class
activation map (CAM) strategy to discover the object regions. However, such
methods only focus on the most discriminative region of the object, resulting
in incomplete segmentation. An alternative is to explore vision transformers
(ViT) to encode the image to acquire the global semantic information. Yet, the
lack of transductive bias to objects is a flaw of ViT. In this paper, we
explore the dual-augmented transformer network with self-regularization
constraints for WSSS. Specifically, we propose a dual network with both
CNN-based and transformer networks for mutually complementary learning, where
both networks augment the final output for enhancement. Massive systemic
evaluations on the challenging PASCAL VOC 2012 benchmark demonstrate the
effectiveness of our method, outperforming previous state-of-the-art methods.

通过使用双增强变换网络和自正则化约束，为弱监督语义分割（WSSS）问题提供一种有效的解决方法，该方法通过将 CNN 和 Transformer 网络相结合进行相互补充学习，并在最终输出上进行增强来改善分割效果。在具有挑战性的 PASCAL VOC 2012 基准测试上进行的大量系统评估结果表明，该方法的有效性优于先前的最先进方法。

双增强变换器网络用于弱监督语义分割

Dual-Augmented Transformer Network for Weakly Supervised Semantic  Segmentation

In this work, we propose a new transformer-based regularization to better
localize objects for Weakly supervised semantic segmentation (WSSS). In
image-level WSSS, Class Activation Map (CAM) is adopted to generate object
localization as pseudo segmentation labels. To address the partial activation
issue of the CAMs, consistency regularization is employed to maintain
activation intensity invariance across various image augmentations. However,
such methods ignore pair-wise relations among regions within each CAM, which
capture context and should also be invariant across image views. To this end,
we propose a new all-pairs consistency regularization (ACR). Given a pair of
augmented views, our approach regularizes the activation intensities between a
pair of augmented views, while also ensuring that the affinity across regions
within each view remains consistent. We adopt vision transformers as the
self-attention mechanism naturally embeds pair-wise affinity. This enables us
to simply regularize the distance between the attention matrices of augmented
image pairs. Additionally, we introduce a novel class-wise localization method
that leverages the gradients of the class token. Our method can be seamlessly
integrated into existing WSSS methods using transformers without modifying the
architectures. We evaluate our method on PASCAL VOC and MS COCO datasets. Our
method produces noticeably better class localization maps (67.3% mIoU on PASCAL
VOC train), resulting in superior WSSS performances.

通过提出新的基于 Transformer 的正则化方法，我们改进了弱监督语义分割（WSSS）中的目标定位问题。我们引入了一种全对一的一致性正则化方法，以保持不同图像增强之间的激活强度不变，并确保在每个 CAM 中的区域之间的关联保持一致。此外，我们还引入了一种利用类令牌梯度的新型类别定位方法。我们的方法可以无缝地集成到使用 Transformer 的现有 WSSS 方法中，并在 PASCAL VOC 和 MS COCO 数据集上得到了卓越的 WSSS 性能。

弱监督语义分割的全对一一致性学习

All-pairs Consistency Learning for Weakly Supervised Semantic  Segmentation

Decisions made by convolutional neural networks(CNN) can be understood and
explained by visualizing discriminative regions on images. To this end, Class
Activation Map (CAM) based methods were proposed as powerful interpretation
tools, making the prediction of deep learning models more explainable,
transparent, and trustworthy. However, all the CAM-based methods (e.g., CAM,
Grad-CAM, and Relevance-CAM) can only be used for interpreting CNN models with
fully-connected (FC) layers as a classifier. It is worth noting that many deep
learning models classify images without FC layers, e.g., few-shot learning
image classification, contrastive learning image classification, and image
retrieval tasks. In this work, a post-hoc interpretation tool named feature
activation map (FAM) is proposed, which can interpret deep learning models
without FC layers as a classifier. In the proposed FAM algorithm, the
channel-wise contribution weights are derived from the similarity scores
between two image embeddings. The activation maps are linearly combined with
the corresponding normalized contribution weights, forming the explanation map
for visualization. The quantitative and qualitative experiments conducted on
ten deep learning models for few-shot image classification, contrastive
learning image classification and image retrieval tasks demonstrate the
effectiveness of the proposed FAM algorithm.

提出了一种名为功能激活映射（Feature Activation Map，FAM）的后期解释工具，以此来解释并理解没有全连接层的深度学习模型（如少样本学习图像分类、对比学习图像分类和图像检索任务），其原理是通过两个图像嵌入之间的相似性得出通道贡献权重，并将激活映射与相应的归一化贡献权重线性组合形成可视化的解释图。经过对十种深度学习模型进行的定量和定性实验表明，该算法的有效性。