Deep reinforcement learning has achieved great successes in recent years,
however, one main challenge is the sample inefficiency. In this paper, we focus
on how to use action guidance by means of a non-expert demonstrator to improve
sample efficiency in a domain with sparse, delayed, and possibly deceptive
rewards: the recently-proposed multi-agent benchmark of Pommerman. We propose a
new framework where even a non-expert simulated demonstrator, e.g., planning
algorithms such as Monte Carlo tree search with a small number rollouts, can be
integrated within asynchronous distributed deep reinforcement learning methods.
Compared to a vanilla deep RL algorithm, our proposed methods both learn faster
and converge to better policies on a two-player mini version of the Pommerman
game.

本文研究如何利用非专家演示者的行动指导来提高 Pommerman 多智能体基准环境下稀疏、延迟和可能具有误导性奖励的领域中的样本效率。我们提出了一个新框架，可以在异步分布式深度强化学习方法中集成非专家演示者，即使用 Monte Carlo 树搜索等策略算法，并以两种玩家为例进行测试，相较于纯粹的深度强化学习算法，我们的提出的方法可以更快地学习，并收敛于更好的策略。

基于 MCTS 的深度强化学习的行动指导

Action Guidance with MCTS for Deep Reinforcement Learning

Safe reinforcement learning has many variants and it is still an open
research problem. Here, we focus on how to use action guidance by means of a
non-expert demonstrator to avoid catastrophic events in a domain with sparse,
delayed, and deceptive rewards: the recently-proposed multi-agent benchmark of
Pommerman. This domain is very challenging for reinforcement learning (RL) ---
past work has shown that model-free RL algorithms fail to achieve significant
learning. In this paper, we shed light into the reasons behind this failure by
exemplifying and analyzing the high rate of catastrophic events (i.e.,
suicides) that happen under random exploration in this domain. While model-free
random exploration is typically futile, we propose a new framework where even a
non-expert simulated demonstrator, e.g., planning algorithms such as Monte
Carlo tree search with small number of rollouts, can be integrated to
asynchronous distributed deep reinforcement learning methods. Compared to
vanilla deep RL algorithms, our proposed methods both learn faster and converge
to better policies on a two-player mini version of the Pommerman game.

本文研究如何使用非专业演示者的行动指导来避免稀疏、延迟和具有欺骗性的奖励域中的灾难性事件，并提出了一种新的框架将计划算法与异步分布式深度强化学习方法相结合，相比传统的深度强化学习算法，提高了学习速度并收敛到更好的策略。