Dynamic 3D interaction has witnessed great interest in recent works, while
creating such 4D content remains challenging. One solution is to animate 3D
scenes with physics-based simulation, and the other is to learn the deformation
of static 3D objects with the distillation of video generative models. The
former one requires assigning precise physical properties to the target object,
otherwise the simulated results would become unnatural. The latter tends to
formulate the video with minor motions and discontinuous frames, due to the
absence of physical constraints in deformation learning. We think that video
generative models are trained with real-world captured data, capable of judging
physical phenomenon in simulation environments. To this end, we propose
DreamPhysics in this work, which estimates physical properties of 3D Gaussian
Splatting with video diffusion priors. DreamPhysics supports both image- and
text-conditioned guidance, optimizing physical parameters via score
distillation sampling with frame interpolation and log gradient. Based on a
material point method simulator with proper physical parameters, our method can
generate 4D content with realistic motions. Experimental results demonstrate
that, by distilling the prior knowledge of video diffusion models, inaccurate
physical properties can be gradually refined for high-quality simulation. Codes
are released at: this https URL

3D 互动和 4D 内容的创作是具有挑战性的，通过动画模拟物理场景或学习静态 3D 物体的变形，结合视频生成模型，我们提出了 DreamPhysics 方法，通过视频扩散先验来估计 3D 高斯光斑的物理属性，以优化物理参数实现逼真的运动，实验证明通过视频扩散模型的先验知识，可以逐渐提高物理模拟的质量。

DreamPhysics: 学习具有视频扩散先验的动态 3D 高斯分布的物理特性

DreamPhysics: Learning Physical Properties of Dynamic 3D Gaussians with  Video Diffusion Priors

Recent years have witnessed significant progress in the field of neural
surface reconstruction. While the extensive focus was put on volumetric and
implicit approaches, a number of works have shown that explicit graphics
primitives such as point clouds can significantly reduce computational
complexity, without sacrificing the reconstructed surface quality. However,
less emphasis has been put on modeling dynamic surfaces with point primitives.
In this work, we present a dynamic point field model that combines the
representational benefits of explicit point-based graphics with implicit
deformation networks to allow efficient modeling of non-rigid 3D surfaces.
Using explicit surface primitives also allows us to easily incorporate
well-established constraints such as-isometric-as-possible regularisation.
While learning this deformation model is prone to local optima when trained in
a fully unsupervised manner, we propose to additionally leverage semantic
information such as keypoint dynamics to guide the deformation learning. We
demonstrate our model with an example application of creating an expressive
animatable human avatar from a collection of 3D scans. Here, previous methods
mostly rely on variants of the linear blend skinning paradigm, which
fundamentally limits the expressivity of such models when dealing with complex
cloth appearances such as long skirts. We show the advantages of our dynamic
point field framework in terms of its representational power, learning
efficiency, and robustness to out-of-distribution novel poses.

本文提出了一种动态点场模型，它将显式点图形的表现优势与隐式形变网络相结合，以便有效地建模非刚性 3D 表面，并演示了该模型在创建表情逼真的可动人体形象方面的优势。