Recent progress in pre-trained diffusion models and 3D generation have
spurred interest in 4D content creation. However, achieving high-fidelity 4D
generation with spatial-temporal consistency remains a challenge. In this work,
we propose STAG4D, a novel framework that combines pre-trained diffusion models
with dynamic 3D Gaussian splatting for high-fidelity 4D generation. Drawing
inspiration from 3D generation techniques, we utilize a multi-view diffusion
model to initialize multi-view images anchoring on the input video frames,
where the video can be either real-world captured or generated by a video
diffusion model. To ensure the temporal consistency of the multi-view sequence
initialization, we introduce a simple yet effective fusion strategy to leverage
the first frame as a temporal anchor in the self-attention computation. With
the almost consistent multi-view sequences, we then apply the score
distillation sampling to optimize the 4D Gaussian point cloud. The 4D Gaussian
spatting is specially crafted for the generation task, where an adaptive
densification strategy is proposed to mitigate the unstable Gaussian gradient
for robust optimization. Notably, the proposed pipeline does not require any
pre-training or fine-tuning of diffusion networks, offering a more accessible
and practical solution for the 4D generation task. Extensive experiments
demonstrate that our method outperforms prior 4D generation works in rendering
quality, spatial-temporal consistency, and generation robustness, setting a new
state-of-the-art for 4D generation from diverse inputs, including text, image,
and video.

在本研究中，我们提出了 STAG4D，这是一种结合了预训练扩散模型和动态 3D 高斯雪碧的新型框架，用于高保真度的 4D 生成，通过利用多视图扩散模型初始化输入视频帧上的多视图图像，以确保多视图序列初始化的时间一致性，并应用得分蒸馏采样来优化 4D 高斯点云。

STAG4D：空间 - 时间定位生成的 4D 高斯

STAG4D: Spatial-Temporal Anchored Generative 4D Gaussians

Aided by text-to-image and text-to-video diffusion models, existing 4D
content creation pipelines utilize score distillation sampling to optimize the
entire dynamic 3D scene. However, as these pipelines generate 4D content from
text or image inputs, they incur significant time and effort in prompt
engineering through trial and error. This work introduces 4DGen, a novel,
holistic framework for grounded 4D content creation that decomposes the 4D
generation task into multiple stages. We identify static 3D assets and
monocular video sequences as key components in constructing the 4D content. Our
pipeline facilitates conditional 4D generation, enabling users to specify
geometry (3D assets) and motion (monocular videos), thus offering superior
control over content creation. Furthermore, we construct our 4D representation
using dynamic 3D Gaussians, which permits efficient, high-resolution
supervision through rendering during training, thereby facilitating
high-quality 4D generation. Additionally, we employ spatial-temporal pseudo
labels on anchor frames, along with seamless consistency priors implemented
through 3D-aware score distillation sampling and smoothness regularizations.
Compared to existing baselines, our approach yields competitive results in
faithfully reconstructing input signals and realistically inferring renderings
from novel viewpoints and timesteps. Most importantly, our method supports
grounded generation, offering users enhanced control, a feature difficult to
achieve with previous methods. Project page:
this https URL

4DGen 是一种新颖的综合框架，将 4D 内容创建任务拆分为多个阶段，利用静态 3D 资产和单目视频序列作为构建 4D 内容的关键组成部分，使用动态 3D Gaussians 构建 4D 表示，在训练期间利用渲染进行高分辨率监督，从而实现了高质量的 4D 生成，并实施了 3D 感知的得分蒸馏采样和平滑正则化以提供空间 - 时间伪标签和无缝一致性先验，相比现有基线方法，在忠实重建输入信号和从新视点和时间步骤推断渲染方面取得了具有竞争力的结果，最重要的是，我们的方法支持基于实际场景的生成，为用户提供了增强控制，这是以前方法难以实现的特点。