This paper proposes a novel transformer-based framework that aims to enhance
weakly supervised semantic segmentation (WSSS) by generating accurate
class-specific object localization maps as pseudo labels. Building upon the
observation that the attended regions of the one-class token in the standard
vision transformer can contribute to a class-agnostic localization map, we
explore the potential of the transformer model to capture class-specific
attention for class-discriminative object localization by learning multiple
class tokens. We introduce a Multi-Class Token transformer, which incorporates
multiple class tokens to enable class-aware interactions with the patch tokens.
To achieve this, we devise a class-aware training strategy that establishes a
one-to-one correspondence between the output class tokens and the ground-truth
class labels. Moreover, a Contrastive-Class-Token (CCT) module is proposed to
enhance the learning of discriminative class tokens, enabling the model to
better capture the unique characteristics and properties of each class. As a
result, class-discriminative object localization maps can be effectively
generated by leveraging the class-to-patch attentions associated with different
class tokens. To further refine these localization maps, we propose the
utilization of patch-level pairwise affinity derived from the patch-to-patch
transformer attention. Furthermore, the proposed framework seamlessly
complements the Class Activation Mapping (CAM) method, resulting in
significantly improved WSSS performance on the PASCAL VOC 2012 and MS COCO 2014
datasets. These results underline the importance of the class token for WSSS.

本文提出了一种新颖的基于 Transformer 的框架，旨在通过生成准确的类特定目标定位图作为伪标签来增强弱监督语义分割，并探讨了 Transformer 模型捕捉类特定关注力以实现具有类鉴别性的目标定位的潜力。

MCTformer+: 弱监督语义分割的多类标记变换器

MCTformer+: Multi-Class Token Transformer for Weakly Supervised Semantic  Segmentation

This paper proposes a new transformer-based framework to learn class-specific
object localization maps as pseudo labels for weakly supervised semantic
segmentation (WSSS). Inspired by the fact that the attended regions of the
one-class token in the standard vision transformer can be leveraged to form a
class-agnostic localization map, we investigate if the transformer model can
also effectively capture class-specific attention for more discriminative
object localization by learning multiple class tokens within the transformer.
To this end, we propose a Multi-class Token Transformer, termed as MCTformer,
which uses multiple class tokens to learn interactions between the class tokens
and the patch tokens. The proposed MCTformer can successfully produce
class-discriminative object localization maps from class-to-patch attentions
corresponding to different class tokens. We also propose to use a patch-level
pairwise affinity, which is extracted from the patch-to-patch transformer
attention, to further refine the localization maps. Moreover, the proposed
framework is shown to fully complement the Class Activation Mapping (CAM)
method, leading to remarkably superior WSSS results on the PASCAL VOC and MS
COCO datasets. These results underline the importance of the class token for
WSSS.

本文提出了一种基于 Transformer 的新框架，用于学习类别特定的物体定位映射，作为弱监督语义分割的伪标签。通过将多个类别令牌引入 Transformer 中，实现对类别特定注意力的有效捕获，从而生成对应于不同类别令牌的类别判别性的物体定位映射，并进一步利用从 patch-patch Transformer 的注意力中提取的 patch-level 成对亲和力来细化定位图。这个提议的框架可以完全补充 Class Activation Mapping（CAM）方法，提高 WSSS 的结果。