We present a method for using previously-trained 'teacher' agents to
kickstart the training of a new 'student' agent. To this end, we leverage ideas
from policy distillation and population based training. Our method places no
constraints on the architecture of the teacher or student agents, and it
regulates itself to allow the students to surpass their teachers in
performance. We show that, on a challenging and computationally-intensive
multi-task benchmark (DMLab-30), kickstarted training improves the data
efficiency of new agents, making it significantly easier to iterate on their
design. We also show that the same kickstarting pipeline can allow a single
student agent to leverage multiple 'expert' teachers which specialize on
individual tasks. In this setting kickstarting yields surprisingly large gains,
with the kickstarted agent matching the performance of an agent trained from
scratch in almost 10x fewer steps, and surpassing its final performance by 42
percent. Kickstarting is conceptually simple and can easily be incorporated
into reinforcement learning experiments.

该论文介绍了一种利用已经训练好的 ' 教师 ' 智能体来启动新的 ' 学生 ' 智能体训练的方法，从而提高新智能体的数据效率，并且可以让单个学生智能体利用多个专门从事不同任务的 ' 专家 ' 教师，获取更好的性能表现。

深度强化学习启动

Kickstarting Deep Reinforcement Learning

Policy advice is a transfer learning method where a student agent is able to
learn faster via advice from a teacher. However, both this and other
reinforcement learning transfer methods have little theoretical analysis. This
paper formally defines a setting where multiple teacher agents can provide
advice to a student and introduces an algorithm to leverage both autonomous
exploration and teacher's advice. Our regret bounds justify the intuition that
good teachers help while bad teachers hurt. Using our formalization, we are
also able to quantify, for the first time, when negative transfer can occur
within such a reinforcement learning setting.

通过多个代理老师的建议，介绍了一种自主探索和老师建议结合的算法，并量化了负面迁移可能发生的情况。