We present algorithms to effectively represent a set of Markov decision
processes (MDPs), whose optimal policies have already been learned, by a
smaller source subset for lifelong, policy-reuse-based transfer learning in
reinforcement learning. This is necessary when the number of previous tasks is
large and the cost of measuring similarity counteracts the benefit of transfer.
The source subset forms an `$\epsilon$-net' over the original set of MDPs, in
the sense that for each previous MDP $M_p$, there is a source $M^s$ whose
optimal policy has $<\epsilon$ regret in $M_p$. Our contributions are as
follows. We present EXP-3-Transfer, a principled policy-reuse algorithm that
optimally reuses a given source policy set when learning for a new MDP. We
present a framework to cluster the previous MDPs to extract a source subset.
The framework consists of (i) a distance $d_V$ over MDPs to measure
policy-based similarity between MDPs; (ii) a cost function $g(\cdot)$ that uses
$d_V$ to measure how good a particular clustering is for generating useful
source tasks for EXP-3-Transfer and (iii) a provably convergent algorithm,
MHAV, for finding the optimal clustering. We validate our algorithms through
experiments in a surveillance domain.

本文提出了一种算法，通过生成一个小且有效的源子集，以实现在强化学习中的全生命周期，基于策略重用的传递学习，从而代表已经学习完最佳策略的一组 MDP。同时，提出了一个聚类的框架，用于提取源子集，并通过在监控领域的实验证实了算法的有效性。