In this paper, we aim to model 3D scene dynamics from multi-view videos.
Unlike the majority of existing works which usually focus on the common task of
novel view synthesis within the training time period, we propose to
simultaneously learn the geometry, appearance, and physical velocity of 3D
scenes only from video frames, such that multiple desirable applications can be
supported, including future frame extrapolation, unsupervised 3D semantic scene
decomposition, and dynamic motion transfer. Our method consists of three major
components, 1) the keyframe dynamic radiance field, 2) the interframe velocity
field, and 3) a joint keyframe and interframe optimization module which is the
core of our framework to effectively train both networks. To validate our
method, we further introduce two dynamic 3D datasets: 1) Dynamic Object
dataset, and 2) Dynamic Indoor Scene dataset. We conduct extensive experiments
on multiple datasets, demonstrating the superior performance of our method over
all baselines, particularly in the critical tasks of future frame extrapolation
and unsupervised 3D semantic scene decomposition.

本文旨在从多视角视频中建模 3D 场景的动态。我们提出了一种新的方法，通过仅利用视频帧学习 3D 场景的几何、外观和物理速度，从而支持多种理想的应用，包括未来帧插值、无监督的 3D 语义场景分解和动态运动传递。