We provide a statistical analysis of regularization-based continual learning
on a sequence of linear regression tasks, with emphasis on how different
regularization terms affect the model performance. We first derive the
convergence rate for the oracle estimator obtained as if all data were
available simultaneously. Next, we consider a family of generalized
$\ell_2$-regularization algorithms indexed by matrix-valued hyperparameters,
which includes the minimum norm estimator and continual ridge regression as
special cases. As more tasks are introduced, we derive an iterative update
formula for the estimation error of generalized $\ell_2$-regularized
estimators, from which we determine the hyperparameters resulting in the
optimal algorithm. Interestingly, the choice of hyperparameters can effectively
balance the trade-off between forward and backward knowledge transfer and
adjust for data heterogeneity. Moreover, the estimation error of the optimal
algorithm is derived explicitly, which is of the same order as that of the
oracle estimator. In contrast, our lower bounds for the minimum norm estimator
and continual ridge regression show their suboptimality. A byproduct of our
theoretical analysis is the equivalence between early stopping and generalized
$\ell_2$-regularization in continual learning, which may be of independent
interest. Finally, we conduct experiments to complement our theory.

我们对基于正则化的连续学习在一系列线性回归任务中进行了统计分析，重点在于不同正则化项如何影响模型性能。我们推导了作为先验估计器的收敛速率，考虑了由矩阵值超参数索引的广义 l2 正则化算法族，包括最小范数估计器和连续岭回归作为特例。随着任务的增加，我们推导了广义 l2 正则化估计器的估计误差的迭代更新公式，从中确定了导致最佳算法的超参数。有趣的是，超参数的选择能够有效平衡前向和后向知识转移的权衡，并适应数据异质性。此外，我们明确地推导出最佳算法的估计误差，它与先验估计器的误差同阶。相比之下，我们的最小范数估计器和连续岭回归的下界显示了它们的子优性。我们的理论分析的副产品是提出了在连续学习中早停和广义 l2 正则化之间的等价性，这可能具有独立的研究价值。最后，我们进行实验以补充我们的理论。