Diffusion models are capable of generating photo-realistic images that combine elements which likely do not appear together in the training set, demonstrating the ability to compositionally generalize. Nonetheless, the precise mechanism of compositionality and how it is acquired through training remains elusive. Inspired by cognitive neuroscientific approaches, we consider a highly reduced setting to examine whether and when diffusion models learn semantically meaningful and factorized representations of composable features. We performed extensive controlled experiments on conditional Denoising Diffusion Probabilistic Models (DDPMs) trained to generate various forms of 2D Gaussian data. We found that the models learn factorized but not fully continuous manifold representations for encoding continuous features of variation underlying the data. With such representations, models demonstrate superior feature compositionality but limited ability to interpolate over unseen values of a given feature. Our experimental results further demonstrate that diffusion models can attain compositionality with few compositional examples, suggesting a more efficient way to train DDPMs. Finally, we connect manifold formation in diffusion models to percolation theory in physics, offering insight into the sudden onset of factorized representation learning. Our thorough toy experiments thus contribute a deeper understanding of how diffusion models capture compositional structure in data.

本研究针对扩散模型在学习语义上有意义的可组合特征表示方面的机制及其训练过程中的挑战进行了探讨。我们通过条件去噪扩散概率模型（DDPMs）进行了大量的控制实验，发现模型能够学习因子化的特征表示，但在对未见值进行插值时能力有限，表明扩散模型可以通过较少的组合示例实现组合性。我们的研究还将扩散模型中的流形形成与物理学中的渗流理论相联系，提供了对因子化表示学习突现的深入理解。 

扩散模型如何学习因子化和组合