Controllable image synthesis models allow creation of diverse images based on
text instructions or guidance from a reference image. Recently, denoising
diffusion probabilistic models have been shown to generate more realistic
imagery than prior methods, and have been successfully demonstrated in
unconditional and class-conditional settings. We investigate fine-grained,
continuous control of this model class, and introduce a novel unified framework
for semantic diffusion guidance, which allows either language or image
guidance, or both. Guidance is injected into a pretrained unconditional
diffusion model using the gradient of image-text or image matching scores,
without re-training the diffusion model. We explore CLIP-based language
guidance as well as both content and style-based image guidance in a unified
framework. Our text-guided synthesis approach can be applied to datasets
without associated text annotations. We conduct experiments on FFHQ and LSUN
datasets, and show results on fine-grained text-guided image synthesis,
synthesis of images related to a style or content reference image, and examples
with both textual and image guidance.

该研究探讨了图像合成模型的细粒度、连续控制，提出了一种新的语义扩散引导统一框架，可以注入预训练的无条件扩散模型的语言或图像指导，并在 FFHQ 和 LSUN 数据集上进行了实验。