Visual pre-training with large-scale real-world data has made great progress
in recent years, showing great potential in robot learning with pixel
observations. However, the recipes of visual pre-training for robot
manipulation tasks are yet to be built. In this paper, we thoroughly
investigate the effects of visual pre-training strategies on robot manipulation
tasks from three fundamental perspectives: pre-training datasets, model
architectures and training methods. Several significant experimental findings
are provided that are beneficial for robot learning. Further, we propose a
visual pre-training scheme for robot manipulation termed Vi-PRoM, which
combines self-supervised learning and supervised learning. Concretely, the
former employs contrastive learning to acquire underlying patterns from
large-scale unlabeled data, while the latter aims learning visual semantics and
temporal dynamics. Extensive experiments on robot manipulations in various
simulation environments and the real robot demonstrate the superiority of the
proposed scheme. Videos and more details can be found on
https://explore-pretrain-robot.github.io.

基于大规模真实世界数据的视觉预训练在最近几年取得了很大的进展，展示了在像素观察中进行机器人学习的巨大潜力。本文从三个基本角度（预训练数据集、模型架构和训练方法）深入研究了视觉预训练策略对机器人操作任务的影响，并提供了几个重要的实验发现，有益于机器人学习。此外，我们提出了一种用于机器人操作的视觉预训练方案，称为 Vi-PRoM，它结合了自监督学习和监督学习。具体而言，前者利用对比学习从大规模无标签数据中获取潜在模式，而后者旨在学习视觉语义和时间动态。在各种仿真环境和真实机器人中进行了大量实验证明了所提出方案的优越性。更多细节和视频可在 https://explore-pretrain-robot.github.io 找到。