3D scene representation for robot manipulation should capture three key
object properties: permanency -- objects that become occluded over time
continue to exist; amodal completeness -- objects have 3D occupancy, even if
only partial observations are available; spatiotemporal continuity -- the
movement of each object is continuous over space and time. In this paper, we
introduce 3D Dynamic Scene Representation (DSR), a 3D volumetric scene
representation that simultaneously discovers, tracks, reconstructs objects, and
predicts their dynamics while capturing all three properties. We further
propose DSR-Net, which learns to aggregate visual observations over multiple
interactions to gradually build and refine DSR. Our model achieves
state-of-the-art performance in modeling 3D scene dynamics with DSR on both
simulated and real data. Combined with model predictive control, DSR-Net
enables accurate planning in downstream robotic manipulation tasks such as
planar pushing. Video is available at this https URL

介绍了一种通过同时发现、跟踪、重构对象及预测其动态来捕获三个关键对象属性的三维动态场景表示方法 DSR，基于此提出了 DS 核网络 DSR-Net 模型，在建立和优化 DSR 模型的过程中逐渐聚合视觉观测，并在仿真和实际数据上实现了 3D 场景动力学建模的最新性能，结合模型预测控制，也为下游机器人操作任务如平面推动提供了准确规划的能力。