The transfer of knowledge from one policy to another is an important tool in
Deep Reinforcement Learning. This process, referred to as distillation, has
been used to great success, for example, by enhancing the optimisation of
agents, leading to stronger performance faster, on harder domains [26, 32, 5,
8]. Despite the widespread use and conceptual simplicity of distillation, many
different formulations are used in practice, and the subtle variations between
them can often drastically change the performance and the resulting objective
that is being optimised. In this work, we rigorously explore the entire
landscape of policy distillation, comparing the motivations and strengths of
each variant through theoretical and empirical analysis. Our results point to
three distillation techniques, that are preferred depending on specifics of the
task. Specifically a newly proposed expected entropy regularised distillation
allows for quicker learning in a wide range of situations, while still
guaranteeing convergence.

本文探究了深度强化学习中的知识转移工具：策略蒸馏（distillation），比较了各种不同形式并通过理论和实证分析确定了三种最佳蒸馏技术，其中新提出的期望熵正规化蒸馏在广泛的情况下能够保证收敛并加快学习速度。