Compared to on-policy policy gradient techniques, off-policy model-free deep
reinforcement learning (RL) that uses previously gathered data can improve
sampling efficiency. However, off-policy learning becomes challenging when the
discrepancy between the distributions of the policy of interest and the
policies that collected the data increases. Although the well-studied
importance sampling and off-policy policy gradient techniques were proposed to
compensate for this discrepancy, they usually require a collection of long
trajectories that increases the computational complexity and induce additional
problems such as vanishing/exploding gradients or discarding many useful
experiences. Moreover, their generalization to continuous action domains is
strictly limited as they require action probabilities, which is unsuitable for
deterministic policies. To overcome these limitations, we introduce a novel
policy similarity measure to mitigate the effects of such discrepancy. Our
method offers an adequate single-step off-policy correction without any
probability estimates, and theoretical results show that it can achieve a
contraction mapping with a fixed unique point, which allows "safe" off-policy
learning. An extensive set of empirical results indicate that our algorithm
substantially improves the state-of-the-art and attains higher returns in fewer
steps than the competing methods by efficiently scheduling the learning rate in
Q-learning and policy optimization.

本文研究了基于离线数据的深度强化学习算法，提出了一种新的策略相似度度量方法来提高算法的采样效率和泛化能力，并且证明了该方法可以实现安全的离线学习。实验证明，该方法相较于其他竞争算法在大多数情况下能够更高效地提高学习效率。