Inverse reinforcement learning (IRL) seeks to learn the reward function from expert trajectories, to understand the task for imitation or collaboration thereby removing the need for manual reward engineering. However, IRL in the context of large, high-dimensional problems with unknown dynamics has been particularly challenging. In this paper, we present a new Variational Lower Bound for IRL (VLB-IRL), which is derived under the framework of a probabilistic graphical model with an optimality node. Our method simultaneously learns the reward function and policy under the learned reward function by maximizing the lower bound, which is equivalent to minimizing the reverse Kullback-Leibler divergence between an approximated distribution of optimality given the reward function and the true distribution of optimality given trajectories. This leads to a new IRL method that learns a valid reward function such that the policy under the learned reward achieves expert-level performance on several known domains. Importantly, the method outperforms the existing state-of-the-art IRL algorithms on these domains by demonstrating better reward from the learned policy.

逆向强化学习通过学习专家轨迹的奖励函数，理解模仿或协作任务，从而消除手动奖励设计的需要。本文提出了一种新的变分下界的逆向强化学习方法(VLB-IRL)，通过最大化下界相当于最小化近似分布和真实分布之间的逆Kullback-Leibler散度，同时学习奖励函数和根据所学奖励函数最大化下界来寻找到达专家级性能的策略，该方法在几个已知领域上优于现有的逆向强化学习算法。

逆强化学习的新型变分下界