Human perception reliably identifies movable and immovable parts of 3D
scenes, and completes the 3D structure of objects and background from
incomplete observations. We learn this skill not via labeled examples, but
simply by observing objects move. In this work, we propose an approach that
observes unlabeled multi-view videos at training time and learns to map a
single image observation of a complex scene, such as a street with cars, to a
3D neural scene representation that is disentangled into movable and immovable
parts while plausibly completing its 3D structure. We separately parameterize
movable and immovable scene parts via 2D neural ground plans. These ground
plans are 2D grids of features aligned with the ground plane that can be
locally decoded into 3D neural radiance fields. Our model is trained
self-supervised via neural rendering. We demonstrate that the structure
inherent to our disentangled 3D representation enables a variety of downstream
tasks in street-scale 3D scenes using simple heuristics, such as extraction of
object-centric 3D representations, novel view synthesis, instance segmentation,
and 3D bounding box prediction, highlighting its value as a backbone for
data-efficient 3D scene understanding models. This disentanglement further
enables scene editing via object manipulation such as deletion, insertion, and
rigid-body motion.

本文提出一种无监督学习方法，通过观察未标记的多视角视频，学习将一个包含多种物体的复杂场景的单幅图像观察映射到一个三维神经场景表示，可以将该表示分解为可移动和不可移动的部分，并通过神经渲染进行自监督训练，从而实现基于对象的三维表示、新视角合成、实例分割和三维边界框预测等多种下游任务，并通过对象操作（如删除、插入和刚体运动）实现场景编辑。