For vision-language models (VLMs), understanding the dynamic properties of
objects and their interactions within 3D scenes from video is crucial for
effective reasoning. In this work, we introduce a video question answering
dataset SuperCLEVR-Physics that focuses on the dynamics properties of objects.
We concentrate on physical concepts -- velocity, acceleration, and collisions
within 4D scenes, where the model needs to fully understand these dynamics
properties and answer the questions built on top of them. From the evaluation
of a variety of current VLMs, we find that these models struggle with
understanding these dynamic properties due to the lack of explicit knowledge
about the spatial structure in 3D and world dynamics in time variants. To
demonstrate the importance of an explicit 4D dynamics representation of the
scenes in understanding world dynamics, we further propose NS-4Dynamics, a
Neural-Symbolic model for reasoning on 4D Dynamics properties under explicit
scene representation from videos. Using scene rendering likelihood combining
physical prior distribution, the 4D scene parser can estimate the dynamics
properties of objects over time to and interpret the observation into 4D scene
representation as world states. By further incorporating neural-symbolic
reasoning, our approach enables advanced applications in future prediction,
factual reasoning, and counterfactual reasoning. Our experiments show that our
NS-4Dynamics suppresses previous VLMs in understanding the dynamics properties
and answering questions about factual queries, future prediction, and
counterfactual reasoning. Moreover, based on the explicit 4D scene
representation, our model is effective in reconstructing the 4D scenes and
re-simulate the future or counterfactual events.

该研究提出了一个基于视频的问答数据集 SuperCLEVR-Physics，专注于对象的动力学特性，通过显式的 4D 场景表示和神经符号推理，展示了 NS-4Dynamics 在理解动力学属性以及未来预测和反事实推理方面的效果，该模型在重建 4D 场景并重新模拟未来或反事实事件方面表现良好。

基于物理先验的组合式 4D 动态场景理解与视频问答

Compositional 4D Dynamic Scenes Understanding with Physics Priors for  Video Question Answering

We introduce VividDream, a method for generating explorable 4D scenes with
ambient dynamics from a single input image or text prompt. VividDream first
expands an input image into a static 3D point cloud through iterative
inpainting and geometry merging. An ensemble of animated videos is then
generated using video diffusion models with quality refinement techniques and
conditioned on renderings of the static 3D scene from the sampled camera
trajectories. We then optimize a canonical 4D scene representation using an
animated video ensemble, with per-video motion embeddings and visibility masks
to mitigate inconsistencies. The resulting 4D scene enables free-view
exploration of a 3D scene with plausible ambient scene dynamics. Experiments
demonstrate that VividDream can provide human viewers with compelling 4D
experiences generated based on diverse real images and text prompts.

用单一的输入图像或文本提示生成可以探索的带有环境动态的 4D 场景的方法。通过迭代修复和几何融合将输入图像扩展为静态的 3D 点云，然后使用质量细化技术并以从采样的相机轨迹渲染的静态 3D 场景为条件，生成带有动态视频集合的动画视频集合。最终优化包含动画视频集合的规范化 4D 场景表示，使用每个视频的运动嵌入和可见性蒙版来减轻不一致性，从而实现 3D 场景的自由视角探索，并且实验证明 VividDream 可以根据不同的真实图像和文本提示为人类观众提供引人入胜的 4D 体验。

VividDream: 生成带有环境动力学的 3D 场景

VividDream: Generating 3D Scene with Ambient Dynamics

Reconstructing 4D scenes from video inputs is a crucial yet challenging task.
Conventional methods usually rely on the assumptions of multi-view video
inputs, known camera parameters, or static scenes, all of which are typically
absent under in-the-wild scenarios. In this paper, we relax all these
constraints and tackle a highly ambitious but practical task, which we termed
as AnyV4D: we assume only one monocular video is available without any camera
parameters as input, and we aim to recover the dynamic 4D world alongside the
camera poses. To this end, we introduce GFlow, a new framework that utilizes
only 2D priors (depth and optical flow) to lift a video (3D) to a 4D explicit
representation, entailing a flow of Gaussian splatting through space and time.
GFlow first clusters the scene into still and moving parts, then applies a
sequential optimization process that optimizes camera poses and the dynamics of
3D Gaussian points based on 2D priors and scene clustering, ensuring fidelity
among neighboring points and smooth movement across frames. Since dynamic
scenes always introduce new content, we also propose a new pixel-wise
densification strategy for Gaussian points to integrate new visual content.
Moreover, GFlow transcends the boundaries of mere 4D reconstruction; it also
enables tracking of any points across frames without the need for prior
training and segments moving objects from the scene in an unsupervised way.
Additionally, the camera poses of each frame can be derived from GFlow,
allowing for rendering novel views of a video scene through changing camera
pose. By employing the explicit representation, we may readily conduct
scene-level or object-level editing as desired, underscoring its versatility
and power. Visit our project website at: this https URL

使用 GFlow 框架，我们从单目视频中恢复动态的 4D 世界和相机姿态，通过使用深度和光流的 2D 先验进行聚类、优化和像素密集化，使得 GFlow 不仅可以进行 4D 重建，还可以进行对象追踪和无监督场景编辑。