Generative pre-trained models have demonstrated remarkable effectiveness in
language and vision domains by learning useful representations. In this paper,
we extend the scope of this effectiveness by showing that visual robot
manipulation can significantly benefit from large-scale video generative
pre-training. We introduce GR-1, a straightforward GPT-style model designed for
multi-task language-conditioned visual robot manipulation. GR-1 takes as inputs
a language instruction, a sequence of observation images, and a sequence of
robot states. It predicts robot actions as well as future images in an
end-to-end manner. Thanks to a flexible design, GR-1 can be seamlessly
finetuned on robot data after pre-trained on a large-scale video dataset. We
perform extensive experiments on the challenging CALVIN benchmark and a real
robot. On CALVIN benchmark, our method outperforms state-of-the-art baseline
methods and improves the success rate from 88.9% to 94.9%. In the setting of
zero-shot unseen scene generalization, GR-1 improves the success rate from
53.3% to 85.4%. In real robot experiments, GR-1 also outperforms baseline
methods and shows strong potentials in generalization to unseen scenes and
objects. We provide inaugural evidence that a unified GPT-style transformer,
augmented with large-scale video generative pre-training, exhibits remarkable
generalization to multi-task visual robot manipulation. Project page:
this https URL

通过大规模视频生成预训练，我们展示了基于语言条件的视觉机器人操作对于生成预训练模型的有效性扩展，提供了新的证据，显示出在多任务视觉机器人操作中，经过视频生成预训练的统一 GPT 风格转换器具有显著的泛化能力。

大规模视频生成预训练在视觉机器人操作中的应用

Unleashing Large-Scale Video Generative Pre-training for Visual Robot  Manipulation

We demonstrate model-based, visual robot manipulation of linear deformable
objects. Our approach is based on a state-space representation of the physical
system that the robot aims to control. This choice has multiple advantages,
including the ease of incorporating physics priors in the dynamics model and
perception model, and the ease of planning manipulation actions. In addition,
physical states can naturally represent object instances of different
appearances. Therefore, dynamics in the state space can be learned in one
setting and directly used in other visually different settings. This is in
contrast to dynamics learned in pixel space or latent space, where
generalization to visual differences are not guaranteed. Challenges in taking
the state-space approach are the estimation of the high-dimensional state of a
deformable object from raw images, where annotations are very expensive on real
data, and finding a dynamics model that is both accurate, generalizable, and
efficient to compute. We are the first to demonstrate self-supervised training
of rope state estimation on real images, without requiring expensive
annotations. This is achieved by our novel self-supervising learning objective,
which is generalizable across a wide range of visual appearances. With
estimated rope states, we train a fast and differentiable neural network
dynamics model that encodes the physics of mass-spring systems. Our method has
a higher accuracy in predicting future states compared to models that do not
involve explicit state estimation and do not use any physics prior, while only
using 3\% of training data. We also show that our approach achieves more
efficient manipulation, both in simulation and on a real robot, when used
within a model predictive controller.

本文提出了一种基于状态空间表示和自我监督训练技术的视觉机器人操作方法，通过估计可变形物体状态、训练神经网络动力学模型以及采用模型预测控制器，实现高效而准确的物体操纵。