This paper proposes a new optimization objective for value-based deep
reinforcement learning. We extend conventional Deep Q-Networks (DQNs) by adding
a model-learning component yielding a transcoder network. The prediction errors
for the model are included in the basic DQN loss as additional regularizers.
This augmented objective leads to a richer training signal that provides
feedback at every time step. Moreover, because learning an environment model
shares a common structure with the RL problem, we hypothesize that the
resulting objective improves both sample efficiency and performance. We
empirically confirm our hypothesis on a range of 20 games from the Atari
benchmark attaining superior results over vanilla DQN without model-based
regularization.

本文提出了一种基于价值的深度强化学习的新的优化目标，通过增加模型学习组件来扩展传统的 DQN，以产生一个转码器网络，学习环境模型和强化学习问题共享结构，该模型的预测误差被包含在基本 DQN 损失中作为额外的正则化项，改进了采样效率和性能。在 Atari 基准测试的 20 个游戏中，我们实验证明了我们的假设，获得了比基础 DQN 更好的结果。