Weakly supervised semantic segmentation has witnessed great achievements with
image-level labels. Several recent approaches use the CLIP model to generate
pseudo labels for training an individual segmentation model, while there is no
attempt to apply the CLIP model as the backbone to directly segment objects
with image-level labels. In this paper, we propose WeCLIP, a CLIP-based
single-stage pipeline, for weakly supervised semantic segmentation.
Specifically, the frozen CLIP model is applied as the backbone for semantic
feature extraction, and a new decoder is designed to interpret extracted
semantic features for final prediction. Meanwhile, we utilize the above frozen
backbone to generate pseudo labels for training the decoder. Such labels cannot
be optimized during training. We then propose a refinement module (RFM) to
rectify them dynamically. Our architecture enforces the proposed decoder and
RFM to benefit from each other to boost the final performance. Extensive
experiments show that our approach significantly outperforms other approaches
with less training cost. Additionally, our WeCLIP also obtains promising
results for fully supervised settings. The code is available at
this https URL

本文提出了基于 CLIP 模型的 WeCLIP 方法，用于弱监督语义分割。WeCLIP 将冻结的 CLIP 模型作为骨干网络进行语义特征提取，并设计了新的解码器来解释提取的语义特征进行最终预测。同时，我们利用冻结的骨干网络生成伪标签来训练解码器，并提出了一个校正模块来动态修正这些标签。我们的架构使得解码器和校正模块相互受益，以提高最终性能。大量实验证明了我们的方法在训练成本较低的情况下明显优于其他方法。此外，我们的 WeCLIP 方法在全监督设置下也取得了有希望的结果。