The ability to learn continually is essential in a complex and changing
world. In this paper, we characterize the behavior of canonical value-based
deep reinforcement learning (RL) approaches under varying degrees of
non-stationarity. In particular, we demonstrate that deep RL agents lose their
ability to learn good policies when they cycle through a sequence of Atari 2600
games. This phenomenon is alluded to in prior work under various guises --
e.g., loss of plasticity, implicit under-parameterization, primacy bias, and
capacity loss. We investigate this phenomenon closely at scale and analyze how
the weights, gradients, and activations change over time in several experiments
with varying dimensions (e.g., similarity between games, number of games,
number of frames per game), with some experiments spanning 50 days and 2
billion environment interactions. Our analysis shows that the activation
footprint of the network becomes sparser, contributing to the diminishing
gradients. We investigate a remarkably simple mitigation strategy --
Concatenated ReLUs (CReLUs) activation function -- and demonstrate its
effectiveness in facilitating continual learning in a changing environment.

探讨在 Atari 2600 游戏环境中使用基于价值的强化学习方法在面对不同程度的非随机性（non-stationarity）时，其所采用的策略的丧失学习能力现象，通过对不同维度条件下的实验结果，发现这种现象与网络中激活函数的稀疏性相关，本文提出的嵌入化 ReLU (CReLUs) 激活函数可以有效提高在环境变化中的持续学习能力。