Foundation models pre-trained on massive unlabeled datasets have revolutionized natural language and computer vision, exhibiting remarkable generalization capabilities, thus highlighting the importance of pre-training. Yet, efforts in robotics have struggled to achieve similar success, limited by either the need for costly robotic annotations or the lack of representations that effectively model the physical world. In this paper, we introduce ARM4R, an Auto-regressive Robotic Model that leverages low-level 4D Representations learned from human video data to yield a better pre-trained robotic model. Specifically, we focus on utilizing 3D point tracking representations from videos derived by lifting 2D representations into 3D space via monocular depth estimation across time. These 4D representations maintain a shared geometric structure between the points and robot state representations up to a linear transformation, enabling efficient transfer learning from human video data to low-level robotic control. Our experiments show that ARM4R can transfer efficiently from human video data to robotics and consistently improves performance on tasks across various robot environments and configurations.

本研究解决了机器人领域中缺乏有效表示和昂贵标注数据的问题，通过引入ARM4R自回归机器人模型，采用从人类视频中学习的低级4D表示进行预训练。这种方法利用从视频中获得的3D点追踪表示，实现了从人类视频数据到低级机器人控制的高效迁移学习，实验表明ARM4R在各种机器人环境和配置下的任务性能都有所提升。

基于4D表示的自回归机器人模型预训练