The remarkable prowess of diffusion models in image generation has spurred efforts to extend their application beyond generative tasks. However, a persistent challenge exists in lacking a unified approach to apply diffusion models to visual perception tasks with diverse semantic granularity requirements. Our purpose is to establish a unified visual perception framework, capitalizing on the potential synergies between generative and discriminative models. In this paper, we propose Vermouth, a simple yet effective framework comprising a pre-trained Stable Diffusion (SD) model containing rich generative priors, a unified head (U-head) capable of integrating hierarchical representations, and an adapted expert providing discriminative priors. Comprehensive investigations unveil potential characteristics of Vermouth, such as varying granularity of perception concealed in latent variables at distinct time steps and various U-net stages. We emphasize that there is no necessity for incorporating a heavyweight or intricate decoder to transform diffusion models into potent representation learners. Extensive comparative evaluations against tailored discriminative models showcase the efficacy of our approach on zero-shot sketch-based image retrieval (ZS-SBIR), few-shot classification, and open-vocabulary semantic segmentation tasks. The promising results demonstrate the potential of diffusion models as formidable learners, establishing their significance in furnishing informative and robust visual representations.

Vermouth是一个简单而有效的框架，由预训练的稳定扩散（SD）模型、能够集成分层表示的统一头部（U-head）和提供鉴别先验的调整专家构成，通过广泛的比较评估，在零样本基于草图的图像检索（ZS-SBIR）、少样本分类和开放词汇语义分割任务上展示了我们方法的效果，展示了扩散模型作为强大学习器的潜力，证明了它们在提供信息丰富和鲁棒的视觉表示方面的重要性。

利用扩散先验将生成模型和判别模型融合为统一的视觉感知模型