We show that deep reinforcement learning can maintain its ability to learn without resetting network parameters in settings where the number of gradient updates greatly exceeds the number of environment samples. Under such large update-to-data ratios, a recent study by Nikishin et al. (2022) suggested the emergence of a primacy bias, in which agents overfit early interactions and downplay later experience, impairing their ability to learn. In this work, we dissect the phenomena underlying the primacy bias. We inspect the early stages of training that ought to cause the failure to learn and find that a fundamental challenge is a long-standing acquaintance: value overestimation. Overinflated Q-values are found not only on out-of-distribution but also in-distribution data and can be traced to unseen action prediction propelled by optimizer momentum. We employ a simple unit-ball normalization that enables learning under large update ratios, show its efficacy on the widely used dm_control suite, and obtain strong performance on the challenging dog tasks, competitive with model-based approaches. Our results question, in parts, the prior explanation for sub-optimal learning due to overfitting on early data.

通过对深度强化学习的分析，我们发现在大量梯度更新次数远远超过环境样本数量的情况下，存在一种优先偏见现象，即代理程序过度依赖早期的交互并低估后期经验，从而影响其学习能力。我们发现这种现象的根本挑战在于价值过高估计，这不仅体现在样本外分布数据上，也表现在样本内分布数据上，并可追溯到由优化器动量推动的未知动作预测。我们采用了一种简单的单位球归一化方法，使得在大量梯度更新比例下学习变得可行，并在广泛使用的dm_control套件上取得了强大的性能，在具有挑战性的dog任务上与基于模型的方法相媲美。我们的结果对于早期数据过拟合导致学习不佳的先前解释提出了部分质疑。

高更新比例下的深度强化学习剖析：克服价值高估和发散