Deep Reinforcement Learning has yielded proficient controllers for complex
tasks. However, these controllers have limited memory and rely on being able to
perceive the complete game screen at each decision point. To address these
shortcomings, this article investigates the effects of adding recurrency to a
Deep Q-Network (DQN) by replacing the first post-convolutional fully-connected
layer with a recurrent LSTM. The resulting \textit{Deep Recurrent Q-Network}
(DRQN), although capable of seeing only a single frame at each timestep,
successfully integrates information through time and replicates DQN's
performance on standard Atari games and partially observed equivalents
featuring flickering game screens. Additionally, when trained with partial
observations and evaluated with incrementally more complete observations,
DRQN's performance scales as a function of observability. Conversely, when
trained with full observations and evaluated with partial observations, DRQN's
performance degrades less than DQN's. Thus, given the same length of history,
recurrency is a viable alternative to stacking a history of frames in the DQN's
input layer and while recurrency confers no systematic advantage when learning
to play the game, the recurrent net can better adapt at evaluation time if the
quality of observations changes.

本文介绍了一种新型深度强化学习模型 Deep Recurrent Q-Network (DRQN)，使用 recurrent LSTM 替换 DQN 的第一个后卷积全连接层，DRQN 在每个决策点只看到一个帧，但可以成功地通过时间积分信息，并且在标准的 Atari 游戏和部分不完整的游戏中表现出与 DQN 相似的性能，且在不同可观察性情况下 DRQN 的性能也随之变化。因此，recurrency 是 DQN 的一种可替代方式。