Leveraging semantically precise pseudo masks derived from image-level class
knowledge for segmentation, namely image-level Weakly Supervised Semantic
Segmentation (WSSS), still remains challenging. While Class Activation Maps
(CAMs) using CNNs have steadily been contributing to the success of WSSS, the
resulting activation maps often narrowly focus on class-specific parts (e.g.,
only face of human). On the other hand, recent works based on vision
transformers (ViT) have shown promising results based on their self-attention
mechanism to capture the semantic parts but fail in capturing complete
class-specific details (e.g., entire body parts of human but also with a dog
nearby). In this work, we propose Complementary Branch (CoBra), a novel dual
branch framework consisting of two distinct architectures which provide
valuable complementary knowledge of class (from CNN) and semantic (from ViT) to
each branch. In particular, we learn Class-Aware Projection (CAP) for the CNN
branch and Semantic-Aware Projection (SAP) for the ViT branch to explicitly
fuse their complementary knowledge and facilitate a new type of extra
patch-level supervision. Our model, through CoBra, fuses CNN and ViT's
complementary outputs to create robust pseudo masks that integrate both class
and semantic information effectively. Extensive experiments qualitatively and
quantitatively investigate how CNN and ViT complement each other on the PASCAL
VOC 2012 dataset, showing a state-of-the-art WSSS result. This includes not
only the masks generated by our model, but also the segmentation results
derived from utilizing these masks as pseudo labels.

在这项工作中，我们提出了一种新的双分支框架 ——CoBra，它由两种不同的架构组成，为每个分支提供有价值的类别（来自 CNN）和语义（来自 ViT）的互补知识，以显式融合它们的补充知识，并促进一种新型的额外模块级监督。通过 CoBra，我们的模型将 CNN 和 ViT 的互补输出融合在一起，有效地融合了类别和语义信息，从而创建出具有鲁棒性的伪掩模，该模型在 PASCAL VOC 2012 数据集上表现出了最先进的弱监督语义分割结果。

CoBra: 强大的弱监督语义分割的补充分支结合类别和语义知识

CoBra: Complementary Branch Fusing Class and Semantic Knowledge for  Robust Weakly Supervised Semantic Segmentation

The way that information propagates in neural networks is of great
importance. In this paper, we propose Path Aggregation Network (PANet) aiming
at boosting information flow in proposal-based instance segmentation framework.
Specifically, we enhance the entire feature hierarchy with accurate
localization signals in lower layers by bottom-up path augmentation, which
shortens the information path between lower layers and topmost feature. We
present adaptive feature pooling, which links feature grid and all feature
levels to make useful information in each feature level propagate directly to
following proposal subnetworks. A complementary branch capturing different
views for each proposal is created to further improve mask prediction. These
improvements are simple to implement, with subtle extra computational overhead.
Our PANet reaches the 1st place in the COCO 2017 Challenge Instance
Segmentation task and the 2nd place in Object Detection task without
large-batch training. It is also state-of-the-art on MVD and Cityscapes. Code
is available at this https URL

本文提出 Path Aggregation Network（PANet）来提高神经网络中信息传递的效率，主要集中于基于提案的实例分割框架。具体来说，通过底层路径增强和自适应特征池化等方法来提高信息的传递效率，并建立了一个补充分支，以有效提高 Mask 预测的准确性。我们的方法在多个数据集上达到了最佳表现，具有简单易实现以及较小的计算复杂度。