One of the great promises of robot learning systems is that they will be able
to learn from their mistakes and continuously adapt to ever-changing
environments. Despite this potential, most of the robot learning systems today
are deployed as a fixed policy and they are not being adapted after their
deployment. Can we efficiently adapt previously learned behaviors to new
environments, objects and percepts in the real world? In this paper, we present
a method and empirical evidence towards a robot learning framework that
facilitates continuous adaption. In particular, we demonstrate how to adapt
vision-based robotic manipulation policies to new variations by fine-tuning via
off-policy reinforcement learning, including changes in background, object
shape and appearance, lighting conditions, and robot morphology. Further, this
adaptation uses less than 0.2% of the data necessary to learn the task from
scratch. We find that our approach of adapting pre-trained policies leads to
substantial performance gains over the course of fine-tuning, and that
pre-training via RL is essential: training from scratch or adapting from
supervised ImageNet features are both unsuccessful with such small amounts of
data. We also find that these positive results hold in a limited continual
learning setting, in which we repeatedly fine-tune a single lineage of policies
using data from a succession of new tasks. Our empirical conclusions are
consistently supported by experiments on simulated manipulation tasks, and by
52 unique fine-tuning experiments on a real robotic grasping system pre-trained
on 580,000 grasps.

本论文提出了一种通过强化学习进行增量式 fine-tuning 的方法，可以有效地将图像为基础的机器人操作策略适应到新的环境、物体和感知中，在不到数据学习任务的 0.2％的情况下实现适应，这种方式可以大幅提高任务的性能表现，并且在连续学习场景下仍保持一致稳定。