Robots will experience non-stationary environment dynamics throughout their
lifetime: the robot dynamics can change due to wear and tear, or its
surroundings may change over time. Eventually, the robots should perform well
in all of the environment variations it has encountered. At the same time, it
should still be able to learn fast in a new environment. We identify two
challenges in Reinforcement Learning (RL) under such a lifelong learning
setting with off-policy data: first, existing off-policy algorithms struggle
with the trade-off between being conservative to maintain good performance in
the old environment and learning efficiently in the new environment, despite
keeping all the data in the replay buffer. We propose the Offline Distillation
Pipeline to break this trade-off by separating the training procedure into an
online interaction phase and an offline distillation phase.Second, we find that
training with the imbalanced off-policy data from multiple environments across
the lifetime creates a significant performance drop. We identify that this
performance drop is caused by the combination of the imbalanced quality and
size among the datasets which exacerbate the extrapolation error of the
Q-function. During the distillation phase, we apply a simple fix to the issue
by keeping the policy closer to the behavior policy that generated the data. In
the experiments, we demonstrate these two challenges and the proposed solutions
with a simulated bipedal robot walk-ing task across various environment
changes. We show that the Offline Distillation Pipeline achieves better
performance across all the encountered environments without affecting data
collection. We also provide a comprehensive empirical study to support our
hypothesis on the data imbalance issue.

本文介绍了在生命周期内，机器人应该如何快速适应不断变化的环境，在强化学习领域下提出了离线蒸馏管道算法，解决了传统算法在新旧环境中表现的困境以及在多种环境中训练数据失衡等问题，并通过模拟仿生机器人步行任务的实验进行了检验。