We explore the task of embodied view synthesis from monocular videos of
deformable scenes. Given a minute-long RGBD video of people interacting with
their pets, we render the scene from novel camera trajectories derived from
in-scene motion of actors: (1) egocentric cameras that simulate the point of
view of a target actor and (2) 3rd-person cameras that follow the actor.
Building such a system requires reconstructing the root-body and articulated
motion of each actor in the scene, as well as a scene representation that
supports free-viewpoint synthesis. Longer videos are more likely to capture the
scene from diverse viewpoints (which helps reconstruction) but are also more
likely to contain larger motions (which complicates reconstruction). To address
these challenges, we present Total-Recon, the first method to
photorealistically reconstruct deformable scenes from long monocular RGBD
videos. Crucially, to scale to long videos, our method hierarchically
decomposes the scene motion into the motion of each object, which itself is
decomposed into global root-body motion and local articulations. To quantify
such "in-the-wild" reconstruction and view synthesis, we collect ground-truth
data from a specialized stereo RGBD capture rig for 11 challenging videos,
significantly outperforming prior art. Code, videos, and data can be found at
this https URL .

本文提出了 Total-Recon 方法，用于从长时间单目 RGBD 视频中重建变形场景并进行自由视点合成，通过场景运动层次分解，实现了快速高效的实现，并最终超越以往的方法。