Inverse Reinforcement Learning (IRL) is a powerful paradigm for inferring a reward function from expert demonstrations. Many IRL algorithms require a known transition model and sometimes even a known expert policy, or they at least require access to a generative model. However, these assumptions are too strong for many real-world applications, where the environment can be accessed only through sequential interaction. We propose a novel IRL algorithm: Active exploration for Inverse Reinforcement Learning (AceIRL), which actively explores an unknown environment and expert policy to quickly learn the expert's reward function and identify a good policy. AceIRL uses previous observations to construct confidence intervals that capture plausible reward functions and find exploration policies that focus on the most informative regions of the environment. AceIRL is the first approach to active IRL with sample-complexity bounds that does not require a generative model of the environment. AceIRL matches the sample complexity of active IRL with a generative model in the worst case. Additionally, we establish a problem-dependent bound that relates the sample complexity of AceIRL to the suboptimality gap of a given IRL problem. We empirically evaluate AceIRL in simulations and find that it significantly outperforms more naive exploration strategies.

本文提出使用主动探索策略的逆强化学习算法（AceIRL），该算法通过构造置信区间捕捉潜在的奖励函数，寻找信息最充分的环境区域的探索策略，从而快速学习专家的奖励函数和制定一个良好的策略。AceIRL是第一种不需要环境生成模型并具有样本复杂度界限的主动逆强化学习方法，并与具备环境生成模型情况下的样本复杂度相匹配，在模拟实验中证明AceIRL优于其他探索策略。

逆强化学习的主动探索