In continual learning, a learner has to keep learning from the data over its whole life time. A key issue is to decide what knowledge to keep and what knowledge to let go. In a neural network, this can be implemented by using a step-size vector to scale how much gradient samples change network weights. Common algorithms, like RMSProp and Adam, use heuristics, specifically normalization, to adapt this step-size vector. In this paper, we show that those heuristics ignore the effect of their adaptation on the overall objective function, for example by moving the step-size vector away from better step-size vectors. On the other hand, stochastic meta-gradient descent algorithms, like IDBD (Sutton, 1992), explicitly optimize the step-size vector with respect to the overall objective function. On simple problems, we show that IDBD is able to consistently improve step-size vectors, where RMSProp and Adam do not. We explain the differences between the two approaches and their respective limitations. We conclude by suggesting that combining both approaches could be a promising future direction to improve the performance of neural networks in continual learning.

在这篇论文中，我们展示了常用的算法（如RMSProp和Adam）在调整步长向量时忽略了其适应过程对整体目标函数的影响，并通过简单问题的实验显示，与RMSProp和Adam相比，IDBD算法可以持续改进步长向量。我们讨论了两种方法的差异和各自的限制，并建议将两种方法结合起来以提高神经网络在持续学习中的性能。

连续学习的步长优化