Reinforcement learning (RL) provides a theoretical framework for continuously improving an agent's behavior via trial and error. However, efficiently learning policies from scratch can be very difficult, particularly for tasks with exploration challenges. In such settings, it might be desirable to initialize RL with an existing policy, offline data, or demonstrations. However, naively performing such initialization in RL often works poorly, especially for value-based methods. In this paper, we present a meta algorithm that can use offline data, demonstrations, or a pre-existing policy to initialize an RL policy, and is compatible with any RL approach. In particular, we propose Jump-Start Reinforcement Learning (JSRL), an algorithm that employs two policies to solve tasks: a guide-policy, and an exploration-policy. By using the guide-policy to form a curriculum of starting states for the exploration-policy, we are able to efficiently improve performance on a set of simulated robotic tasks. We show via experiments that JSRL is able to significantly outperform existing imitation and reinforcement learning algorithms, particularly in the small-data regime. In addition, we provide an upper bound on the sample complexity of JSRL and show that with the help of a guide-policy, one can improve the sample complexity for non-optimism exploration methods from exponential in horizon to polynomial.

本文提出了 JSRL 算法，该算法通过使用先导策略（guide-policy）和探索策略（exploration-policy）来解决机器人任务，从而实现了通过离线数据、演示或现有策略来初始化RL策略，并能够兼容任何RL方法。通过实现，我们展示了JSRL 能够在小数据时代优于现有的模仿和强化学习算法，并且我们提供了JSRL样本复杂性的上限，并证明了在有了先导策略的情形下，可以将传统的样本复杂性从指数级降至多项式。

强化学习入门