Recent research has explored the utilization of pre-trained text-image
discriminative models, such as CLIP, to tackle the challenges associated with
open-vocabulary semantic segmentation. However, it is worth noting that the
alignment process based on contrastive learning employed by these models may
unintentionally result in the loss of crucial localization information and
object completeness, which are essential for achieving accurate semantic
segmentation. More recently, there has been an emerging interest in extending
the application of diffusion models beyond text-to-image generation tasks,
particularly in the domain of semantic segmentation. These approaches utilize
diffusion models either for generating annotated data or for extracting
features to facilitate semantic segmentation. This typically involves training
segmentation models by generating a considerable amount of synthetic data or
incorporating additional mask annotations. To this end, we uncover the
potential of generative text-to-image conditional diffusion models as highly
efficient open-vocabulary semantic segmenters, and introduce a novel
training-free approach named DiffSegmenter. Specifically, by feeding an input
image and candidate classes into an off-the-shelf pre-trained conditional
latent diffusion model, the cross-attention maps produced by the denoising
U-Net are directly used as segmentation scores, which are further refined and
completed by the followed self-attention maps. Additionally, we carefully
design effective textual prompts and a category filtering mechanism to further
enhance the segmentation results. Extensive experiments on three benchmark
datasets show that the proposed DiffSegmenter achieves impressive results for
open-vocabulary semantic segmentation.

最近的研究探讨了利用预训练的文本 - 图像判别模型（如 CLIP）来解决与开放词汇语义分割相关的挑战。然而，值得注意的是，这些模型基于对比学习的对齐过程可能无意中导致重要的定位信息和物体完整性的丢失，这对于实现准确的语义分割至关重要。最近，人们越来越有兴趣将扩散模型应用于文本到图像生成任务以外的领域，特别是在语义分割领域。这些方法利用扩散模型生成注释数据或提取特征以促进语义分割。为此，我们揭示了生成式文本到图像条件扩散模型作为高效的开放词汇语义分割器的潜力，并引入了一种名为 DiffSegmenter 的无需训练的新方法。具体而言，通过将输入图像和候选类别输入到现成的预训练条件潜在扩散模型，由去噪 U-Net 产生的交叉注意力图直接用作分割分数，然后由后续的自注意力图进一步细化和完成分割。此外，我们精心设计了有效的文本提示和类别过滤机制以进一步增强分割结果。在三个基准数据集上的广泛实验证明，所提出的 DiffSegmenter 在开放词汇语义分割方面取得了令人印象深刻的结果。