The reinforcement learning (RL) problem is rife with sources of
non-stationarity, making it a notoriously difficult problem domain for the
application of neural networks. We identify a mechanism by which non-stationary
prediction targets can prevent learning progress in deep RL agents:
\textit{capacity loss}, whereby networks trained on a sequence of target values
lose their ability to quickly update their predictions over time. We
demonstrate that capacity loss occurs in a range of RL agents and environments,
and is particularly damaging to performance in sparse-reward tasks. We then
present a simple regularizer, Initial Feature Regularization (InFeR), that
mitigates this phenomenon by regressing a subspace of features towards its
value at initialization, leading to significant performance improvements in
sparse-reward environments such as Montezuma's Revenge. We conclude that
preventing capacity loss is crucial to enable agents to maximally benefit from
the learning signals they obtain throughout the entire training trajectory.

本文研究深度强化学习中的一种现象 —— 容量丧失，提出 Initial Feature Regularization（InFeR）算法，该算法可以有效缓解容量丧失并显著提高稀疏奖励任务中的性能表现。