We propose behavior-driven optimization via Wasserstein distances (WDs) to improve several classes of state-of-the-art reinforcement learning (RL) algorithms. We show that WD regularizers acting on appropriate policy embeddings efficiently incorporate behavioral characteristics into policy optimization. We demonstrate that they improve Evolution Strategy methods by encouraging more efficient exploration, can be applied in imitation learning and to speed up training of Trust Region Policy Optimization methods. Since the exact computation of WDs is expensive, we develop approximate algorithms based on the combination of different methods: dual formulation of the optimal transport problem, alternating optimization and random feature maps, to effectively replace exact WD computations in the RL tasks considered. We provide theoretical analysis of our algorithms and exhaustive empirical evaluation in a variety of RL settings.

通过使用Wasserstein距离在新定义的潜在行为空间中比较强化学习策略，我们引入了一种新方法，展示了通过使用Wasserstein距离的二元制定，可以学习策略行为的得分函数，用于引导/远离所需/不需要的行为，并将正则化项加入两个新的策略训练算法中。在一系列具有挑战性的环境中展示了比现有方法更好的表现。同时我们提供开源演示。

指导策略优化的行为评分学习