While many recent advances in deep reinforcement learning (RL) rely on
model-free methods, model-based approaches remain an alluring prospect for
their potential to exploit unsupervised data to learn environment model. In
this work, we provide an extensive study on the design of deep generative
models for RL environments and propose a sample efficient and robust method to
learn the model of Atari environments. We deploy this model and propose
generative adversarial tree search (GATS) a deep RL algorithm that learns the
environment model and implements Monte Carlo tree search (MCTS) on the learned
model for planning. While MCTS on the learned model is computationally
expensive, similar to AlphaGo, GATS follows depth limited MCTS. GATS employs
deep Q network (DQN) and learns a Q-function to assign values to the leaves of
the tree in MCTS. We theoretical analyze GATS vis-a-vis the bias-variance
trade-off and show GATS is able to mitigate the worst-case error in the
Q-estimate. While we were expecting GATS to enjoy a better sample complexity
and faster converges to better policies, surprisingly, GATS fails to outperform
DQN. We provide a study on which we show why depth limited MCTS fails to
perform desirably.

本文研究了深度生成模型在强化学习环境中的设计，并提出了一种基于生成对抗树搜索和蒙特卡罗树搜索的算法 GATS。GATS 在学习环境模型后，在树搜索中实现了 MCTS，并使用深度 Q 网络学习 Q 函数。然而，实验结果表明，GATS 并未能超越 DQN，作者分析了深度限制 MCTS 失败的原因。