Following its success in natural language processing and computer vision, foundation models that are pre-trained on large-scale multi-task datasets have also shown great potential in robotics. However, most existing robot foundation models rely solely on 2D image observations, ignoring 3D geometric information, which is essential for robots to perceive and reason about the 3D world. In this paper, we introduce FP3, a first large-scale 3D foundation policy model for robotic manipulation. FP3 builds on a scalable diffusion transformer architecture and is pre-trained on 60k trajectories with point cloud observations. With the model design and diverse pre-training data, FP3 can be efficiently fine-tuned for downstream tasks while exhibiting strong generalization capabilities. Experiments on real robots demonstrate that with only 80 demonstrations, FP3 is able to learn a new task with over 90% success rates in novel environments with unseen objects, significantly surpassing existing robot foundation models.

本研究解决了现有机器人基础模型忽视3D几何信息的问题，提出了一种针对机器人操控的首个大型3D基础策略模型FP3。该模型基于可扩展的扩散变换器架构，并通过60k个点云观测的轨迹进行预训练。实验结果显示，FP3在新环境中以超过90%的成功率学习新任务，显著领先现有的机器人基础模型。

FP3：用于机器人操控的3D基础策略