Given a dataset of expert demonstrations, inverse reinforcement learning
(IRL) aims to recover a reward for which the expert is optimal. This work
proposes a model-free algorithm to solve entropy-regularized IRL problem. In
particular, we employ a stochastic gradient descent update for the reward and a
stochastic soft policy iteration update for the policy. Assuming access to a
generative model, we prove that our algorithm is guaranteed to recover a reward
for which the expert is $\varepsilon$-optimal using
$\mathcal{O}(1/\varepsilon^{2})$ samples of the Markov decision process (MDP).
Furthermore, with $\mathcal{O}(1/\varepsilon^{4})$ samples we prove that the
optimal policy corresponding to the recovered reward is $\varepsilon$-close to
the expert policy in total variation distance.

给定一个专家示范数据集，逆向强化学习（IRL）旨在恢复一个专家所优化的奖励。本研究提出了一种无模型算法来解决熵正则化的 IRL 问题。我们采用随机梯度下降算法更新奖励，并采用随机软策略迭代算法更新策略，假设可以访问一个生成模型，我们证明了我们的算法使用 O (1/ε^2) 个马尔可夫决策过程（MDP）样本能够恢复一个 ε- 最优奖励。此外，我们证明在 O (1/ε^4) 个样本情况下，所恢复的奖励对应的最优策略与专家策略在总变差距离上接近 ε。