We propose Episodic Backward Update (EBU) - a novel deep reinforcement
learning algorithm with a direct value propagation. In contrast to the
conventional use of the experience replay with uniform random sampling, our
agent samples a whole episode and successively propagates the value of a state
to its previous states. Our computationally efficient recursive algorithm
allows sparse and delayed rewards to propagate directly through all transitions
of the sampled episode. We theoretically prove the convergence of the EBU
method and experimentally demonstrate its performance in both deterministic and
stochastic environments. Especially in 49 games of Atari 2600 domain, EBU
achieves the same mean and median human normalized performance of DQN by using
only 5% and 10% of samples, respectively.

本文提出了具有直接价值传播能力的一种新型深度强化学习算法 ——Episodic Backward Update (EBU)。与传统方法通过经验重放的方式使用均匀随机采样不同，我们的算法通过采样整个回合并将状态值连续传递到前一状态。我们的递归算法实现了高效的计算，允许稀疏和延迟奖励直接通过所采样的全部转移。我们在理论上证明了 EBU 方法的收敛性，并在确定性和随机化环境下进行了实验。尤其是在 Atari 2600 领域的 49 个游戏中，EBU 方法仅使用 5% 和 10% 的采样，就能实现与 DQN 相同的平均和中位数人类归一化性能。