Recent breakthroughs in text-to-image synthesis have been driven by diffusion models trained on billions of image-text pairs. Adapting this approach to 3D synthesis would require large-scale datasets of labeled 3D data and efficient architectures for denoising 3D data, neither of which currently exist. In this work, we circumvent these limitations by using a pretrained 2D text-to-image diffusion model to perform text-to-3D synthesis. We introduce a loss based on probability density distillation that enables the use of a 2D diffusion model as a prior for optimization of a parametric image generator. Using this loss in a DeepDream-like procedure, we optimize a randomly-initialized 3D model (a Neural Radiance Field, or NeRF) via gradient descent such that its 2D renderings from random angles achieve a low loss. The resulting 3D model of the given text can be viewed from any angle, relit by arbitrary illumination, or composited into any 3D environment. Our approach requires no 3D training data and no modifications to the image diffusion model, demonstrating the effectiveness of pretrained image diffusion models as priors.

本文提出了一种使用基于文本转图像的扩散模型进行文本到3D合成的方法，该方法绕过了需要大规模标记的3D数据集和能够去噪的3D数据的限制，将2D的扩散模型作为先验，通过梯度下降优化3D模型（Neural Radiance Field），并使用概率密度蒸馏引入的损失函数将2D扩散模型与3D模型相结合。这种方法不需要3D训练数据，也不需要修改图像扩散模型，证明了使用预训练的图像扩散模型作为先验的有效性。

DreamFusion: 使用2D扩散进行文本到3D转换