Learning to imitate expert behavior from demonstrations can be challenging,
especially in environments with high-dimensional, continuous observations and
unknown dynamics. Supervised learning methods based on behavioral cloning (BC)
suffer from distribution shift: because the agent greedily imitates
demonstrated actions, it can drift away from demonstrated states due to error
accumulation. Recent methods based on reinforcement learning (RL), such as
inverse RL and generative adversarial imitation learning (GAIL), overcome this
issue by training an RL agent to match the demonstrations over a long horizon.
Since the true reward function for the task is unknown, these methods learn a
reward function from the demonstrations, often using complex and brittle
approximation techniques that involve adversarial training. We propose a simple
alternative that still uses RL, but does not require learning a reward
function. The key idea is to provide the agent with an incentive to match the
demonstrations over a long horizon, by encouraging it to return to demonstrated
states upon encountering new, out-of-distribution states. We accomplish this by
giving the agent a constant reward of r=+1 for matching the demonstrated action
in a demonstrated state, and a constant reward of r=0 for all other behavior.
Our method, which we call soft Q imitation learning (SQIL), can be implemented
with a handful of minor modifications to any standard Q-learning or off-policy
actor-critic algorithm. Theoretically, we show that SQIL can be interpreted as
a regularized variant of BC that uses a sparsity prior to encourage
long-horizon imitation. Empirically, we show that SQIL outperforms BC and
achieves competitive results compared to GAIL, on a variety of image-based and
low-dimensional tasks in Box2D, Atari, and MuJoCo.

提出了一种称作 “软 Q 模仿学习” 的新方法，该方法使用强化学习，但不需要学习奖励函数，而是通过鼓励智能体在遇到不在分布中的状态时返回演示状态来激励智能体进行长视野模仿。在图像和低维度任务方面表现优于基于行为克隆和生成对抗模仿学习（GAIL）的方法，可用于标准 Q 学习或离线策略演员 - 评论家算法。