Text-to-image diffusion models produce impressive results but are frustrating tools for artists who desire fine-grained control. For example, a common use case is to create images of a specific instance in novel contexts, i.e., "identity-preserving generation". This setting, along with many other tasks (e.g., relighting), is a natural fit for image+text-conditional generative models. However, there is insufficient high-quality paired data to train such a model directly. We propose Diffusion Self-Distillation, a method for using a pre-trained text-to-image model to generate its own dataset for text-conditioned image-to-image tasks. We first leverage a text-to-image diffusion model's in-context generation ability to create grids of images and curate a large paired dataset with the help of a Visual-Language Model. We then fine-tune the text-to-image model into a text+image-to-image model using the curated paired dataset. We demonstrate that Diffusion Self-Distillation outperforms existing zero-shot methods and is competitive with per-instance tuning techniques on a wide range of identity-preservation generation tasks, without requiring test-time optimization.

本研究解决了文本到图像扩散模型在细粒度控制方面的不足，通过提出扩散自蒸馏方法，利用预训练的文本到图像模型生成自己的数据集，以支持文本条件的图像生成任务。研究结果表明，该方法在多种身份保留生成任务上优于现有的零-shot 方法，且与特定实例调优技术具有竞争力，且无需在测试时进行优化。

零-shot定制图像生成的扩散自蒸馏