Existing inverse reinforcement learning methods (e.g. MaxEntIRL, $f$-IRL)
search over candidate reward functions and solve a reinforcement learning
problem in the inner loop. This creates a rather strange inversion where a
harder problem, reinforcement learning, is in the inner loop of a presumably
easier problem, imitation learning. In this work, we show that better
utilization of expert demonstrations can reduce the need for hard exploration
in the inner RL loop, hence accelerating learning. Specifically, we propose two
simple recipes: (1) placing expert transitions into the replay buffer of the
inner RL algorithm (e.g. Soft-Actor Critic) which directly informs the learner
about high reward states instead of forcing the learner to discover them
through extensive exploration, and (2) using expert actions in Q value
bootstrapping in order to improve the target Q value estimates and more
accurately describe high value expert states. Our methods show significant
gains over a MaxEntIRL baseline on the benchmark MuJoCo suite of tasks,
speeding up recovery to 70\% of deterministic expert performance by 2.13x on
HalfCheetah-v2, 2.6x on Ant-v2, 18x on Hopper-v2, and 3.36x on Walker2d-v2.

通过更好地利用专家演示，我们提出了两种简单的方法，即将专家演示数据放入内部强化学习算法的重放缓冲区中以直接通知学习器高奖励状态，以及在 Q 值引导中使用专家操作以改善目标 Q 值估计并更准确地描述高价值专家状态。在 MuJoCo 任务套件中，我们的方法相对于 MaxEntIRL 基准在 HalfCheetah-v2 上将恢复速度提高 2.13 倍，在 Ant-v2 上提高 2.6 倍，在 Hopper-v2 上提高 18 倍，在 Walker2d-v2 上提高 3.36 倍。