Due to their capacity to generate novel and high-quality samples, diffusion
models have attracted significant research interest in recent years. Notably,
the typical training objective of diffusion models, i.e., denoising score
matching, has a closed-form optimal solution that can only generate training
data replicating samples. This indicates that a memorization behavior is
theoretically expected, which contradicts the common generalization ability of
state-of-the-art diffusion models, and thus calls for a deeper understanding.
Looking into this, we first observe that memorization behaviors tend to occur
on smaller-sized datasets, which motivates our definition of effective model
memorization (EMM), a metric measuring the maximum size of training data at
which a learned diffusion model approximates its theoretical optimum. Then, we
quantify the impact of the influential factors on these memorization behaviors
in terms of EMM, focusing primarily on data distribution, model configuration,
and training procedure. Besides comprehensive empirical results identifying the
influential factors, we surprisingly find that conditioning training data on
uninformative random labels can significantly trigger the memorization in
diffusion models. Our study holds practical significance for diffusion model
users and offers clues to theoretical research in deep generative models. Code
is available at this https URL

扩散模型是近年来引起重要研究兴趣的高质量样本生成方法，我们的研究旨在深入理解扩散模型的记忆行为，包括有效模型记忆的定义与影响因素的量化分析。我们的实证研究发现数据分布、模型配置和训练过程等因素对记忆行为有重要影响，并且在扩散模型中，将训练数据与随机标签进行关联显著触发记忆行为。这对扩散模型的使用者具有实际意义，并为深度生成模型的理论研究提供了线索。