We provide a statistical analysis of regularization-based continual learning
on a sequence of linear regression tasks, with emphasis on how different
regularization terms affect the model performance. We first derive the
convergence rate for the oracle estimator obtained as if all data were
available simultaneously. Next, we consider a family of generalized
$\ell_2$-regularization algorithms indexed by matrix-valued hyperparameters,
which includes the minimum norm estimator and continual ridge regression as
special cases. As more tasks are introduced, we derive an iterative update
formula for the estimation error of generalized $\ell_2$-regularized
estimators, from which we determine the hyperparameters resulting in the
optimal algorithm. Interestingly, the choice of hyperparameters can effectively
balance the trade-off between forward and backward knowledge transfer and
adjust for data heterogeneity. Moreover, the estimation error of the optimal
algorithm is derived explicitly, which is of the same order as that of the
oracle estimator. In contrast, our lower bounds for the minimum norm estimator
and continual ridge regression show their suboptimality. A byproduct of our
theoretical analysis is the equivalence between early stopping and generalized
$\ell_2$-regularization in continual learning, which may be of independent
interest. Finally, we conduct experiments to complement our theory.

我们对基于正则化的连续学习在一系列线性回归任务中进行了统计分析，重点在于不同正则化项如何影响模型性能。我们推导了作为先验估计器的收敛速率，考虑了由矩阵值超参数索引的广义 l2 正则化算法族，包括最小范数估计器和连续岭回归作为特例。随着任务的增加，我们推导了广义 l2 正则化估计器的估计误差的迭代更新公式，从中确定了导致最佳算法的超参数。有趣的是，超参数的选择能够有效平衡前向和后向知识转移的权衡，并适应数据异质性。此外，我们明确地推导出最佳算法的估计误差，它与先验估计器的误差同阶。相比之下，我们的最小范数估计器和连续岭回归的下界显示了它们的子优性。我们的理论分析的副产品是提出了在连续学习中早停和广义 l2 正则化之间的等价性，这可能具有独立的研究价值。最后，我们进行实验以补充我们的理论。

基于正则化的持续学习的统计理论

A Statistical Theory of Regularization-Based Continual Learning

Neural networks have achieved remarkable success in many cognitive tasks.
However, when they are trained sequentially on multiple tasks without access to
old data, their performance on early tasks tend to drop significantly. This
problem is often referred to as catastrophic forgetting, a key challenge in
continual learning of neural networks. The regularization-based approach is one
of the primary classes of methods to alleviate catastrophic forgetting. In this
paper, we provide a novel viewpoint of regularization-based continual learning
by formulating it as a second-order Taylor approximation of the loss function
of each task. This viewpoint leads to a unified framework that can be
instantiated to derive many existing algorithms such as Elastic Weight
Consolidation and Kronecker factored Laplace approximation. Based on this
viewpoint, we study the optimization aspects (i.e., convergence) as well as
generalization properties (i.e., finite-sample guarantees) of
regularization-based continual learning. Our theoretical results indicate the
importance of accurate approximation of the Hessian matrix. The experimental
results on several benchmarks provide empirical validation of our theoretical
findings.

本文提出了一种正则化连续学习的新视角，将其定义为每个任务损失函数的二阶 Taylor 近似，得到了一个可实例化的统一框架，并研究了优化和泛化特性，理论和实验结果表明二阶 Hessian 矩阵的精确近似非常重要。