Deep reinforcement learning achieves superhuman performance in a range of
video game environments, but requires that a designer manually specify a reward
function. It is often easier to provide demonstrations of a target behavior
than to design a reward function describing that behavior. Inverse
reinforcement learning (IRL) algorithms can infer a reward from demonstrations
in low-dimensional continuous control environments, but there has been little
work on applying IRL to high-dimensional video games. In our CNN-AIRL baseline,
we modify the state-of-the-art adversarial IRL (AIRL) algorithm to use CNNs for
the generator and discriminator. To stabilize training, we normalize the reward
and increase the size of the discriminator training dataset. We additionally
learn a low-dimensional state representation using a novel autoencoder
architecture tuned for video game environments. This embedding is used as input
to the reward network, improving the sample efficiency of expert
demonstrations. Our method achieves high-level performance on the simple
Catcher video game, substantially outperforming the CNN-AIRL baseline. We also
score points on the Enduro Atari racing game, but do not match expert
performance, highlighting the need for further work.

通过改良强化学习反求解奖励函数模型的 CNN-AIRL 方法以及使用对电子游戏环境进行了调整的自动编码器将专家演示转化为奖励网络的输入，本文在简单的 Catcher 游戏上取得了高水平表现，显著超越了 CNN-AIRL 基线，但在 Enduro Atari 赛车游戏中表现不佳，这彰显了需要进一步研究的必要性。