Existing dynamic scene generation methods mostly rely on distilling knowledge
from pre-trained 3D generative models, which are typically fine-tuned on
synthetic object datasets. As a result, the generated scenes are often
object-centric and lack photorealism. To address these limitations, we
introduce a novel pipeline designed for photorealistic text-to-4D scene
generation, discarding the dependency on multi-view generative models and
instead fully utilizing video generative models trained on diverse real-world
datasets. Our method begins by generating a reference video using the video
generation model. We then learn the canonical 3D representation of the video
using a freeze-time video, delicately generated from the reference video. To
handle inconsistencies in the freeze-time video, we jointly learn a per-frame
deformation to model these imperfections. We then learn the temporal
deformation based on the canonical representation to capture dynamic
interactions in the reference video. The pipeline facilitates the generation of
dynamic scenes with enhanced photorealism and structural integrity, viewable
from multiple perspectives, thereby setting a new standard in 4D scene
generation.

为了解决现有动态场景生成方法的局限性，本研究提出一种新的流程，通过使用视频生成模型，摒弃了对多视图生成模型的依赖，从而充分利用了在多样真实世界数据集上训练的视频生成模型，生成具有增强的逼真度和结构完整性的动态场景，可从多个角度查看，为 4D 场景生成树立了新的标准。