Continual learning aims to learn a series of tasks sequentially without
forgetting the knowledge acquired from the previous ones. In this work, we
propose the Hessian Aware Low-Rank Perturbation algorithm for continual
learning. By modeling the parameter transitions along the sequential tasks with
the weight matrix transformation, we propose to apply the low-rank
approximation on the task-adaptive parameters in each layer of the neural
networks. Specifically, we theoretically demonstrate the quantitative
relationship between the Hessian and the proposed low-rank approximation. The
approximation ranks are then globally determined according to the marginal
increment of the empirical loss estimated by the layer-specific gradient and
low-rank approximation error. Furthermore, we control the model capacity by
pruning less important parameters to diminish the parameter growth. We conduct
extensive experiments on various benchmarks, including a dataset with
large-scale tasks, and compare our method against some recent state-of-the-art
methods to demonstrate the effectiveness and scalability of our proposed
method. Empirical results show that our method performs better on different
benchmarks, especially in achieving task order robustness and handling the
forgetting issue. A demo code can be found at this https URL

本研究提出了一种基于 Hessian 矩阵感知的低秩扰动算法，用于连续学习。通过将参数转换建模为序列任务的权重矩阵转换，我们在神经网络的每一层上应用任务自适应参数的低秩近似。我们在理论上证明了 Hessian 矩阵与提出的低秩近似之间的量化关系，并根据每层梯度和低秩近似误差的经验损失边际增量来全局确定近似秩。此外，我们通过修剪较不重要的参数来控制模型容量，以减少参数增长。我们在各种基准测试上进行广泛实验证明了我们提出方法的有效性和可扩展性，并与一些最新的先进方法进行了比较。实证结果表明，在不同测试中我们的方法表现更好，特别是在实现任务顺序稳健性和处理遗忘问题方面。可在此链接找到演示代码：https:// 此网址连接 。