The issue of generative pretraining for vision models has persisted as a
long-standing conundrum. At present, the text-to-image (T2I) diffusion model
demonstrates remarkable proficiency in generating high-definition images
matching textual inputs, a feat made possible through its pre-training on
large-scale image-text pairs. This leads to a natural inquiry: can diffusion
models be utilized to tackle visual perception tasks? In this paper, we propose
a simple yet effective scheme to harness a diffusion model for visual
perception tasks. Our key insight is to introduce learnable embeddings (meta
prompts) to the pre-trained diffusion models to extract proper features for
perception. The effect of meta prompts are two-fold. First, as a direct
replacement of the text embeddings in the T2I models, it can activate
task-relevant features during feature extraction. Second, it will be used to
re-arrange the extracted features to ensures that the model focuses on the most
pertinent features for the task on hand. Additionally, we design a recurrent
refinement training strategy that fully leverages the property of diffusion
models, thereby yielding stronger visual features. Extensive experiments across
various benchmarks validate the effectiveness of our approach. Our approach
achieves new performance records in depth estimation tasks on NYU depth V2 and
KITTI, and in semantic segmentation task on CityScapes. Concurrently, the
proposed method attains results comparable to the current state-of-the-art in
semantic segmentation on ADE20K and pose estimation on COCO datasets, further
exemplifying its robustness and versatility.

通过引入可学习的嵌入（元提示）来利用扩散模型解决视觉感知任务，我们的方法在深度估计和语义分割任务上取得了新的性能记录，并在 ADE20K 的语义分割和 COCO 数据集的姿态估计等方面达到了与最先进方法相媲美的结果，展示了其稳健性和多功能性。