The recent wave of large-scale text-to-image diffusion models has
dramatically increased our text-based image generation abilities. These models
can generate realistic images for a staggering variety of prompts and exhibit
impressive compositional generalization abilities. Almost all use cases thus
far have solely focused on sampling; however, diffusion models can also provide
conditional density estimates, which are useful for tasks beyond image
generation. In this paper, we show that the density estimates from large-scale
text-to-image diffusion models like Stable Diffusion can be leveraged to
perform zero-shot classification without any additional training. Our
generative approach to classification attains strong results on a variety of
benchmarks and outperforms alternative methods of extracting knowledge from
diffusion models. We also find that our diffusion-based approach has stronger
multimodal relational reasoning abilities than competing contrastive
approaches. Finally, we evaluate diffusion models trained on ImageNet and find
that they approach the performance of SOTA discriminative classifiers trained
on the same dataset, even with weak augmentations and no regularization.
Results and visualizations at this https URL

本文介绍了基于文本和图像结合模型的生成分类方法，利用如 Stable Diffusion 的扩展图像扩散模型提供的在不需要额外训练的条件概率密度估计，实现了零样本分类，并证明其能够在多种基准测试上获得强大的结果及优于竞争方法的效果，同时还发现与对比差异对比性方法相比，扩散方法在多模式关系推理方面具有更强的能力，并且即使在弱增强和没有正则化的情况下，基于图像的扩散模型也能取得与训练于同一数据集的 SOTA 判别分类器相近的性能。