Deep reinforcement learning has emerged as a powerful tool for a variety of learning tasks, however deep nets typically exhibit forgetting when learning multiple tasks in sequence. To mitigate forgetting, we propose an experience replay process that augments the standard FIFO buffer and selectively stores experiences in a long-term memory. We explore four strategies for selecting which experiences will be stored: favoring surprise, favoring reward, matching the global training distribution, and maximizing coverage of the state space. We show that distribution matching successfully prevents catastrophic forgetting, and is consistently the best approach on all domains tested. While distribution matching has better and more consistent performance, we identify one case in which coverage maximization is beneficial - when tasks that receive less trained are more important. Overall, our results show that selective experience replay, when suitable selection algorithms are employed, can prevent catastrophic forgetting.

本文研究通过经验重放的方法，利用“distribution matching”算法来防止深度神经网络多任务学习中的严重忘却问题。实验探索了不同体验选择策略的优缺点，结果表明分布匹配算法可以成功避免严重忘却问题，并且在所有测试领域中始终表现最佳。同时我们发现，当部分任务的重要性比其他任务更高时，最大化状态空间覆盖率是有福利的。

选择性经验回放用于终身学习