Temporal difference (TD) learning is often used to update the estimate of the
value function which is used by RL agents to extract useful policies. In this
paper, we focus on value function estimation in continual reinforcement
learning. We propose to decompose the value function into two components which
update at different timescales: a permanent value function, which holds general
knowledge that persists over time, and a transient value function, which allows
quick adaptation to new situations. We establish theoretical results showing
that our approach is well suited for continual learning and draw connections to
the complementary learning systems (CLS) theory from neuroscience. Empirically,
this approach improves performance significantly on both prediction and control
problems.

本文提出了将值函数分解为两个组成部分的方法，分别在不同的时间尺度上更新，其中永久性值函数持有随时间持久存在的一般知识，而短暂性值函数允许快速适应新情况，理论结果表明这种方法非常适用于连续学习，并与神经科学中的互补学习系统（CLS）理论建立了联系，实证结果表明该方法在预测和控制问题上显著提高性能。