Recent works have shown that visual pretraining on egocentric datasets using
masked autoencoders (MAE) can improve generalization for downstream robotics
tasks. However, these approaches pretrain only on 2D images, while many
robotics applications require 3D scene understanding. In this work, we propose
3D-MVP, a novel approach for 3D multi-view pretraining using masked
autoencoders. We leverage Robotic View Transformer (RVT), which uses a
multi-view transformer to understand the 3D scene and predict gripper pose
actions. We split RVT's multi-view transformer into visual encoder and action
decoder, and pretrain its visual encoder using masked autoencoding on
large-scale 3D datasets such as Objaverse. We evaluate 3D-MVP on a suite of
virtual robot manipulation tasks and demonstrate improved performance over
baselines. We also show promising results on a real robot platform with minimal
finetuning. Our results suggest that 3D-aware pretraining is a promising
approach to improve sample efficiency and generalization of vision-based
robotic manipulation policies. We will release code and pretrained models for
3D-MVP to facilitate future research. Project site:
this https URL

提出了 3D-MVP，一种使用遮罩自编码器进行 3D 多视图预训练的新方法，通过在大规模 3D 数据集上使用遮罩自编码器对其视觉编码器进行预训练，评估了其在虚拟机器人操作任务中的性能并展示了超过基线的改进，同时在实际机器人平台上也取得了令人期待的结果。