This paper presents a novel paradigm for building scalable 3D generative
models utilizing pre-trained video diffusion models. The primary obstacle in
developing foundation 3D generative models is the limited availability of 3D
data. Unlike images, texts, or videos, 3D data are not readily accessible and
are difficult to acquire. This results in a significant disparity in scale
compared to the vast quantities of other types of data. To address this issue,
we propose using a video diffusion model, trained with extensive volumes of
text, images, and videos, as a knowledge source for 3D data. By unlocking its
multi-view generative capabilities through fine-tuning, we generate a
large-scale synthetic multi-view dataset to train a feed-forward 3D generative
model. The proposed model, VFusion3D, trained on nearly 3M synthetic multi-view
data, can generate a 3D asset from a single image in seconds and achieves
superior performance when compared to current SOTA feed-forward 3D generative
models, with users preferring our results over 70% of the time.

本论文提出了一种利用预训练视频扩散模型构建可扩展的 3D 生成模型的新范例。通过使用视频扩散模型作为 3D 数据的知识源，通过微调其多视图生成能力，生成一个大规模的合成多视图数据集，从而训练出一个前馈的 3D 生成模型。该模型在近 300 万个合成多视图数据上训练，能够在几秒钟内从单张图像生成一个 3D 资源，并在与当前最先进的前馈 3D 生成模型相比取得了更好的性能，用户 70％的时间更喜欢我们的结果。