Pre-trained diffusion models have demonstrated remarkable proficiency in
synthesizing images across a wide range of scenarios with customizable prompts,
indicating their effective capacity to capture universal features. Motivated by
this, our study delves into the utilization of the implicit knowledge embedded
within diffusion models to address challenges in cross-domain semantic
segmentation. This paper investigates the approach that leverages the sampling
and fusion techniques to harness the features of diffusion models efficiently.
Contrary to the simplistic migration applications characterized by prior
research, our finding reveals that the multi-step diffusion process inherent in
the diffusion model manifests more robust semantic features. We propose
DIffusion Feature Fusion (DIFF) as a backbone use for extracting and
integrating effective semantic representations through the diffusion process.
By leveraging the strength of text-to-image generation capability, we introduce
a new training framework designed to implicitly learn posterior knowledge from
it. Through rigorous evaluation in the contexts of domain generalization
semantic segmentation, we establish that our methodology surpasses preceding
approaches in mitigating discrepancies across distinct domains and attains the
state-of-the-art (SOTA) benchmark. Within the synthetic-to-real (syn-to-real)
context, our method significantly outperforms ResNet-based and
transformer-based backbone methods, achieving an average improvement of
$3.84\%$ mIoU across various datasets. The implementation code will be released
soon.

利用预训练扩散模型的内在知识，通过采样和融合技术，提出了一种新的扩散特征融合方法（DIFF）来提取和集成有效的语义表示，该方法在领域泛化语义分割和合成到真实情景中均优于先前的方法达到了最优性能。

通过扩散特征弥合语义分割的领域差距

Diffusion Features to Bridge Domain Gap for Semantic Segmentation

Prototypical contrastive learning (PCL) has been widely used to learn
class-wise domain-invariant features recently. These methods are based on the
assumption that the prototypes, which are represented as the central value of
the same class in a certain domain, are domain-invariant. Since the prototypes
of different domains have discrepancies as well, the class-wise
domain-invariant features learned from the source domain by PCL need to be
aligned with the prototypes of other domains simultaneously. However, the
prototypes of the same class in different domains may be different while the
prototypes of different classes may be similar, which may affect the learning
of class-wise domain-invariant features. Based on these observations, a
calibration-based dual prototypical contrastive learning (CDPCL) approach is
proposed to reduce the domain discrepancy between the learned class-wise
features and the prototypes of different domains for domain generalization
semantic segmentation. It contains an uncertainty-guided PCL (UPCL) and a
hard-weighted PCL (HPCL). Since the domain discrepancies of the prototypes of
different classes may be different, we propose an uncertainty probability
matrix to represent the domain discrepancies of the prototypes of all the
classes. The UPCL estimates the uncertainty probability matrix to calibrate the
weights of the prototypes during the PCL. Moreover, considering that the
prototypes of different classes may be similar in some circumstances, which
means these prototypes are hard-aligned, the HPCL is proposed to generate a
hard-weighted matrix to calibrate the weights of the hard-aligned prototypes
during the PCL. Extensive experiments demonstrate that our approach achieves
superior performance over current approaches on domain generalization semantic
segmentation tasks.

提出了一种基于校准的双原型对比学习方法，用于减少学习类别无关特征与不同域原型之间的域差异，对于域泛化语义分割，通过不确定性引导的对比学习和加权对比学习来校准原型权重，实验证明该方法在域泛化语义分割任务上具有优越性能。