Text-to-image diffusion models have shown powerful ability on conditional
image synthesis. With large-scale vision-language pre-training, diffusion
models are able to generate high-quality images with rich texture and
reasonable structure under different text prompts. However, it is an open
problem to adapt the pre-trained diffusion model for visual perception. In this
paper, we propose an implicit and explicit language guidance framework for
diffusion-based perception, named IEDP. Our IEDP comprises of an implicit
language guidance branch and an explicit language guidance branch. The implicit
branch employs frozen CLIP image encoder to directly generate implicit text
embeddings that are fed to diffusion model, without using explicit text
prompts. The explicit branch utilizes the ground-truth labels of corresponding
images as text prompts to condition feature extraction of diffusion model.
During training, we jointly train diffusion model by sharing the model weights
of these two branches. As a result, implicit and explicit branches can jointly
guide feature learning. During inference, we only employ implicit branch for
final prediction, which does not require any ground-truth labels. Experiments
are performed on two typical perception tasks, including semantic segmentation
and depth estimation. Our IEDP achieves promising performance on both tasks.
For semantic segmentation, our IEDP has the mIoU score of 55.9% on AD20K
validation set, which outperforms the baseline method VPD by 2.2%. For depth
estimation, our IEDP outperforms the baseline method VPD with a relative gain
of 10.2%.

本研究介绍了一种基于隐式和显式语言引导的 IGDP 扩散感知框架，利用大规模视觉 - 语言预训练实现高质量图像生成，同时在语义分割和深度估计等感知任务上取得了有希望的性能。