Pre-trained diffusion models have demonstrated remarkable proficiency in
synthesizing images across a wide range of scenarios with customizable prompts,
indicating their effective capacity to capture universal features. Motivated by
this, our study delves into the utilization of the implicit knowledge embedded
within diffusion models to address challenges in cross-domain semantic
segmentation. This paper investigates the approach that leverages the sampling
and fusion techniques to harness the features of diffusion models efficiently.
Contrary to the simplistic migration applications characterized by prior
research, our finding reveals that the multi-step diffusion process inherent in
the diffusion model manifests more robust semantic features. We propose
DIffusion Feature Fusion (DIFF) as a backbone use for extracting and
integrating effective semantic representations through the diffusion process.
By leveraging the strength of text-to-image generation capability, we introduce
a new training framework designed to implicitly learn posterior knowledge from
it. Through rigorous evaluation in the contexts of domain generalization
semantic segmentation, we establish that our methodology surpasses preceding
approaches in mitigating discrepancies across distinct domains and attains the
state-of-the-art (SOTA) benchmark. Within the synthetic-to-real (syn-to-real)
context, our method significantly outperforms ResNet-based and
transformer-based backbone methods, achieving an average improvement of
$3.84\%$ mIoU across various datasets. The implementation code will be released
soon.

利用预训练扩散模型的内在知识，通过采样和融合技术，提出了一种新的扩散特征融合方法（DIFF）来提取和集成有效的语义表示，该方法在领域泛化语义分割和合成到真实情景中均优于先前的方法达到了最优性能。

通过扩散特征弥合语义分割的领域差距

Diffusion Features to Bridge Domain Gap for Semantic Segmentation

In unsupervised domain adaptation (UDA), a model trained on source data (e.g.
synthetic) is adapted to target data (e.g. real-world) without access to target
annotation. Most previous UDA methods struggle with classes that have a similar
visual appearance on the target domain as no ground truth is available to learn
the slight appearance differences. To address this problem, we propose a Masked
Image Consistency (MIC) module to enhance UDA by learning spatial context
relations of the target domain as additional clues for robust visual
recognition. MIC enforces the consistency between predictions of masked target
images, where random patches are withheld, and pseudo-labels that are generated
based on the complete image by an exponential moving average teacher. To
minimize the consistency loss, the network has to learn to infer the
predictions of the masked regions from their context. Due to its simple and
universal concept, MIC can be integrated into various UDA methods across
different visual recognition tasks such as image classification, semantic
segmentation, and object detection. MIC significantly improves the
state-of-the-art performance across the different recognition tasks for
synthetic-to-real, day-to-nighttime, and clear-to-adverse-weather UDA. For
instance, MIC achieves an unprecedented UDA performance of 75.9 mIoU and 92.8%
on GTA-to-Cityscapes and VisDA-2017, respectively, which corresponds to an
improvement of +2.1 and +3.0 percent points over the previous state of the art.
The implementation is available at this https URL

本文提出了一种新的被称为 Masked Image Consistency (MIC) 的模块，用于提高无监督域适应（UDA）的性能，它通过学习目标域中的空间上下文关系作为强健视觉识别的附加线索来加强 UDA。该方法可用于不同的视觉识别任务，如图像分类，语义分割和对象检测，在合成到真实、日间到夜间、以及清晰到逆境天气的 UDA 中，显著提高了最新技术的表现。

MIC: 用于上下文增强领域自适应的遮蔽图像一致性方法

MIC: Masked Image Consistency for Context-Enhanced Domain Adaptation

We tackle the problem of unsupervised synthetic-to-real domain adaptation for
single image depth estimation. An essential building block of single image
depth estimation is an encoder-decoder task network that takes RGB images as
input and produces depth maps as output. In this paper, we propose a novel
training strategy to force the task network to learn domain invariant
representations in a selfsupervised manner. Specifically, we extend
self-supervised learning from traditional representation learning, which works
on images from a single domain, to domain invariant representation learning,
which works on images from two different domains by utilizing an image-to-image
translation network. Firstly, we use an image-to-image translation network to
transfer domain-specific styles between synthetic and real domains. This style
transfer operation allows us to obtain similar images from the different
domains. Secondly, we jointly train our task network and Siamese network with
the same images from the different domains to obtain domain invariance for the
task network. Finally, we fine-tune the task network using labeled synthetic
and unlabeled realworld data. Our training strategy yields improved
generalization capability in the real-world domain. We carry out an extensive
evaluation on two popular datasets for depth estimation, KITTI and Make3D. The
results demonstrate that our proposed method outperforms the state-of-the-art
on all metrics, e.g. by 14.7% on Sq Rel on KITTI. The source code and model
weights will be made available.

本文提出一种自监督学习的新训练策略，通过图像转换网络实现在合成和真实领域之间的域不变表示学习，从而提高单张图像深度估计在现实世界中的泛化能力。实验结果表明，该方法在 KITTI 和 Make3D 数据集上均优于现有技术。