Continual learning, focused on sequentially learning multiple tasks, has
gained significant attention recently. Despite the tremendous progress made in
the past, the theoretical understanding, especially factors contributing to
catastrophic forgetting, remains relatively unexplored. In this paper, we
provide a general theoretical analysis of forgetting in the linear regression
model via Stochastic Gradient Descent (SGD) applicable to both
underparameterized and overparameterized regimes. Our theoretical framework
reveals some interesting insights into the intricate relationship between task
sequence and algorithmic parameters, an aspect not fully captured in previous
studies due to their restrictive assumptions. Specifically, we demonstrate
that, given a sufficiently large data size, the arrangement of tasks in a
sequence, where tasks with larger eigenvalues in their population data
covariance matrices are trained later, tends to result in increased forgetting.
Additionally, our findings highlight that an appropriate choice of step size
will help mitigate forgetting in both underparameterized and overparameterized
settings. To validate our theoretical analysis, we conducted simulation
experiments on both linear regression models and Deep Neural Networks (DNNs).
Results from these simulations substantiate our theoretical findings.

通过随机梯度下降（SGD）在线性回归模型中提供了一般性的理论分析，揭示了任务序列与算法参数之间错综复杂的关系，探讨了连续学习中的灾难性遗忘和算法参数选取对任务学习顺序的影响，通过模拟实验验证了理论分析的结果。

线性回归中连续学习中遗忘的理解

Understanding Forgetting in Continual Learning with Linear Regression

The Alternating Direction Method of Multipliers (ADMM) has gained significant
attention across a broad spectrum of machine learning applications.
Incorporating the over-relaxation technique shows potential for enhancing the
convergence rate of ADMM. However, determining optimal algorithmic parameters,
including both the associated penalty and relaxation parameters, often relies
on empirical approaches tailored to specific problem domains and contextual
scenarios. Incorrect parameter selection can significantly hinder ADMM's
convergence rate. To address this challenge, in this paper we first propose a
general approach to optimize the value of penalty parameter, followed by a
novel closed-form formula to compute the optimal relaxation parameter in the
context of linear quadratic problems (LQPs). We then experimentally validate
our parameter selection methods through random instantiations and diverse
imaging applications, encompassing diffeomorphic image registration, image
deblurring, and MRI reconstruction.

在这篇论文中，我们首先提出了一种通用方法来优化惩罚参数的值，然后在线性二次问题（LQPs）的背景下，提出了一种新的封闭形式公式来计算最佳松弛参数。通过随机实例化和多样化的图像应用实验证明了我们参数选择方法的有效性，其中包括差分同胚图像配准、图像去模糊和 MRI 重建。