As text-conditioned diffusion models (DMs) achieve breakthroughs in image,
video, and 3D generation, the research community's focus has shifted to the
more challenging task of text-to-4D synthesis, which introduces a temporal
dimension to generate dynamic 3D objects. In this context, we identify Score
Distillation Sampling (SDS), a widely used technique for text-to-3D synthesis,
as a significant hindrance to text-to-4D performance due to its Janus-faced and
texture-unrealistic problems coupled with high computational costs. In this
paper, we propose \textbf{P}ixel-\textbf{L}evel \textbf{A}lignments for
Text-to-\textbf{4D} Gaussian Splatting (\textbf{PLA4D}), a novel method that
utilizes text-to-video frames as explicit pixel alignment targets to generate
static 3D objects and inject motion into them. Specifically, we introduce Focal
Alignment to calibrate camera poses for rendering and GS-Mesh Contrastive
Learning to distill geometry priors from rendered image contrasts at the pixel
level. Additionally, we develop Motion Alignment using a deformation network to
drive changes in Gaussians and implement Reference Refinement for smooth 4D
object surfaces. These techniques enable 4D Gaussian Splatting to align
geometry, texture, and motion with generated videos at the pixel level.
Compared to previous methods, PLA4D produces synthesized outputs with better
texture details in less time and effectively mitigates the Janus-faced problem.
PLA4D is fully implemented using open-source models, offering an accessible,
user-friendly, and promising direction for 4D digital content creation. Our
project page:
\href{https://github.com/MiaoQiaowei/PLA4D.github.io}{this https URL}.

基于文本的 4D 合成一直是一个具有挑战性的任务，本文提出了 Pixel-Level Alignments for Text-to-4D Gaussian Splatting (PLA4D) 方法，通过使用文本到视频帧作为像素对齐目标，生成静态的 3D 对象，并注入运动，有效解决了 Score Distillation Sampling 技术在文本到 4D 合成中的问题。

PLA4D：用于文字到 4D 高斯雨滴落点的像素级对齐

PLA4D: Pixel-Level Alignments for Text-to-4D Gaussian Splatting

Large-scale diffusion generative models are greatly simplifying image, video
and 3D asset creation from user-provided text prompts and images. However, the
challenging problem of text-to-4D dynamic 3D scene generation with diffusion
guidance remains largely unexplored. We propose Dream-in-4D, which features a
novel two-stage approach for text-to-4D synthesis, leveraging (1) 3D and 2D
diffusion guidance to effectively learn a high-quality static 3D asset in the
first stage; (2) a deformable neural radiance field that explicitly
disentangles the learned static asset from its deformation, preserving quality
during motion learning; and (3) a multi-resolution feature grid for the
deformation field with a displacement total variation loss to effectively learn
motion with video diffusion guidance in the second stage. Through a user
preference study, we demonstrate that our approach significantly advances image
and motion quality, 3D consistency and text fidelity for text-to-4D generation
compared to baseline approaches. Thanks to its motion-disentangled
representation, Dream-in-4D can also be easily adapted for controllable
generation where appearance is defined by one or multiple images, without the
need to modify the motion learning stage. Thus, our method offers, for the
first time, a unified approach for text-to-4D, image-to-4D and personalized 4D
generation tasks.

通过使用扩散生成模型，我们提出了 Dream-in-4D 方法，可以有效地实现从文本和图像生成动态的 3D 场景，该方法包括利用扩散指导学习高质量的静态 3D 资源、使用可变形神经光辐射场将学习的静态资源与形变分离、以及使用多分辨率特征网格和位移总变差损失来学习具有视频扩散指导的运动。通过用户偏好研究，我们证明了与基线方法相比，我们的方法在图像质量、动态一致性和文本保真度方面显著提高了文本到 4D 生成的效果。由于其运动分离表示，我们的方法还可以轻松应用于可控的生成，其中外观由一个或多个图像定义，无需修改运动学习阶段。因此，我们的方法首次提供了一种统一的方法，用于文本到 4D、图像到 4D 和个性化 4D 生成任务。