Deep reinforcement learning methods exhibit impressive performance on a range
of tasks but still struggle on hard exploration tasks in large environments
with sparse rewards. To address this, intrinsic rewards can be generated using
forward model prediction errors that decrease as the environment becomes known,
and incentivize an agent to explore novel states. While prediction-based
intrinsic rewards can help agents solve hard exploration tasks, they can suffer
from catastrophic forgetting and actually increase at visited states. We first
examine the conditions and causes of catastrophic forgetting in grid world
environments. We then propose a new method FARCuriosity, inspired by how humans
and animals learn. The method depends on fragmentation and recall: an agent
fragments an environment based on surprisal, and uses different local curiosity
modules (prediction-based intrinsic reward functions) for each fragment so that
modules are not trained on the entire environment. At each fragmentation event,
the agent stores the current module in long-term memory (LTM) and either
initializes a new module or recalls a previously stored module based on its
match with the current state. With fragmentation and recall, FARCuriosity
achieves less forgetting and better overall performance in games with varied
and heterogeneous environments in the Atari benchmark suite of tasks. Thus,
this work highlights the problem of catastrophic forgetting in prediction-based
curiosity methods and proposes a solution.

在大型环境中，深度强化学习方法在多个任务上表现出色，但在具有稀疏奖励的困难探索任务上仍然面临困难。本研究发现基于预测的内在奖励方法可能出现灾难性遗忘，并提出了一种名为 FARCuriosity 的新方法，通过碎片化和回溯来减轻灾难性遗忘问题，提升了在具有不同环境的游戏中的性能表现。