For autonomous skill acquisition, robots have to learn about the physical
rules governing the 3D world dynamics from their own past experience to predict
and reason about plausible future outcomes. To this end, we propose a
transformation-based 3D video prediction (T3VIP) approach that explicitly
models the 3D motion by decomposing a scene into its object parts and
predicting their corresponding rigid transformations. Our model is fully
unsupervised, captures the stochastic nature of the real world, and the
observational cues in image and point cloud domains constitute its learning
signals. To fully leverage all the 2D and 3D observational signals, we equip
our model with automatic hyperparameter optimization (HPO) to interpret the
best way of learning from them. To the best of our knowledge, our model is the
first generative model that provides an RGB-D video prediction of the future
for a static camera. Our extensive evaluation with simulated and real-world
datasets demonstrates that our formulation leads to interpretable 3D models
that predict future depth videos while achieving on-par performance with 2D
models on RGB video prediction. Moreover, we demonstrate that our model
outperforms 2D baselines on visuomotor control. Videos, code, dataset, and
pre-trained models are available at this http URL.

为了实现机器人的自主技能获取，本文提出了一种基于转化的 3D 视频预测（T3VIP）方法，通过将场景分解为其对象部件并预测其相应的刚性变换，从自身的过去经验中明确地模拟 3D 运动，预测未来的可行结果，并具备自适应超参数优化技术，是首个提供固定摄像头的 RGB-D 视频预测的生成模型。