Diffusion models (DMs) have shown remarkable capabilities in generating
realistic high-quality images, audios, and videos. They benefit significantly
from extensive pre-training on large-scale datasets, including web-crawled data
with paired data and conditions, such as image-text and image-class pairs.
Despite rigorous filtering, these pre-training datasets often inevitably
contain corrupted pairs where conditions do not accurately describe the data.
This paper presents the first comprehensive study on the impact of such
corruption in pre-training data of DMs. We synthetically corrupt ImageNet-1K
and CC3M to pre-train and evaluate over 50 conditional DMs. Our empirical
findings reveal that various types of slight corruption in pre-training can
significantly enhance the quality, diversity, and fidelity of the generated
images across different DMs, both during pre-training and downstream adaptation
stages. Theoretically, we consider a Gaussian mixture model and prove that
slight corruption in the condition leads to higher entropy and a reduced
2-Wasserstein distance to the ground truth of the data distribution generated
by the corruptly trained DMs. Inspired by our analysis, we propose a simple
method to improve the training of DMs on practical datasets by adding condition
embedding perturbations (CEP). CEP significantly improves the performance of
various DMs in both pre-training and downstream tasks. We hope that our study
provides new insights into understanding the data and pre-training processes of
DMs.

通过对扩散模型（DMs）预训练数据的综合研究，发现轻微的破坏性改变可以显著提高 DMs 生成图像的质量和多样性，同时介绍了一种简单的方法 —— 条件嵌入扰动（CEP），可大幅改善 DMs 在预训练和后续任务中的性能。