The ability to reuse previous policies is an important aspect of human
intelligence. To achieve efficient policy reuse, a Deep Reinforcement Learning
(DRL) agent needs to decide when to reuse and which source policies to reuse.
Previous methods solve this problem by introducing extra components to the
underlying algorithm, such as hierarchical high-level policies over source
policies, or estimations of source policies' value functions on the target
task. However, training these components induces either optimization
non-stationarity or heavy sampling cost, significantly impairing the
effectiveness of transfer. To tackle this problem, we propose a novel policy
reuse algorithm called Critic-gUided Policy reuse (CUP), which avoids training
any extra components and efficiently reuses source policies. CUP utilizes the
critic, a common component in actor-critic methods, to evaluate and choose
source policies. At each state, CUP chooses the source policy that has the
largest one-step improvement over the current target policy, and forms a
guidance policy. The guidance policy is theoretically guaranteed to be a
monotonic improvement over the current target policy. Then the target policy is
regularized to imitate the guidance policy to perform efficient policy search.
Empirical results demonstrate that CUP achieves efficient transfer and
significantly outperforms baseline algorithms.

提出了一种名为 Critic-guided Policy 的算法用于处理深度强化学习中的策略重用问题，通过使用评论家来评估并选择源策略，避免了训练额外的组件，该算法在有效的策略搜索方面表现出良好性能，胜过其它基准算法。