Existing on-policy imitation learning algorithms, such as DAgger, assume access to a fixed supervisor. However, there are many settings where the supervisor may converge during policy learning, such as a human performing a novel task or an improving algorithmic controller. We formalize imitation learning from a "converging supervisor" and provide sublinear static and dynamic regret guarantees against the best policy in hindsight with labels from the converged supervisor, even when labels during learning are only from intermediate supervisors. We then show that this framework is closely connected to a recent class of reinforcement learning (RL) algorithms known as dual policy iteration (DPI), which alternate between training a reactive learner with imitation learning and a model-based supervisor with data from the learner. Experiments suggest that when this framework is applied with the state-of-the-art deep model-based RL algorithm PETS as an improving supervisor, it outperforms deep RL baselines on continuous control tasks and provides up to an 80-fold speedup in policy evaluation.

本文讨论了在学习策略时，监督者可能会变化的问题，并提出了一种从收敛监督者进行模仿学习并对其进行形式化。此外，作者将此框架与一类强化学习算法（DPI）相连，并在实验中使用最新的深度模型为监督者的方法在连续控制任务中获得了比深度强化学习基线更好的结果，并提供了多达80倍的策略评估加速。

基于收敛上级的同策略机器人仿真学习