While diffusion models have shown impressive performance in 2D image/video
generation, diffusion-based Text-to-Multi-view-Video (T2MVid) generation
remains underexplored. The new challenges posed by T2MVid generation lie in the
lack of massive captioned multi-view videos and the complexity of modeling such
multi-dimensional distribution. To this end, we propose a novel diffusion-based
pipeline that generates high-quality multi-view videos centered around a
dynamic 3D object from text. Specifically, we factor the T2MVid problem into
viewpoint-space and time components. Such factorization allows us to combine
and reuse layers of advanced pre-trained multi-view image and 2D video
diffusion models to ensure multi-view consistency as well as temporal coherence
for the generated multi-view videos, largely reducing the training cost. We
further introduce alignment modules to align the latent spaces of layers from
the pre-trained multi-view and the 2D video diffusion models, addressing the
reused layers' incompatibility that arises from the domain gap between 2D and
multi-view data. In support of this and future research, we further contribute
a captioned multi-view video dataset. Experimental results demonstrate that our
method generates high-quality multi-view videos, exhibiting vivid motions,
temporal coherence, and multi-view consistency, given a variety of text
prompts.

提出了一种基于扩散模型的新方法，从文本生成以动态三维物体为中心的高质量多视图视频，通过将问题分解为视点空间和时间组件，并引入对齐模块来保证生成的多视图视频的一致性和连贯性，同时解决了二维和多视图数据之间的领域差异。