Continual learning, focused on sequentially learning multiple tasks, has
gained significant attention recently. Despite the tremendous progress made in
the past, the theoretical understanding, especially factors contributing to
catastrophic forgetting, remains relatively unexplored. In this paper, we
provide a general theoretical analysis of forgetting in the linear regression
model via Stochastic Gradient Descent (SGD) applicable to both
underparameterized and overparameterized regimes. Our theoretical framework
reveals some interesting insights into the intricate relationship between task
sequence and algorithmic parameters, an aspect not fully captured in previous
studies due to their restrictive assumptions. Specifically, we demonstrate
that, given a sufficiently large data size, the arrangement of tasks in a
sequence, where tasks with larger eigenvalues in their population data
covariance matrices are trained later, tends to result in increased forgetting.
Additionally, our findings highlight that an appropriate choice of step size
will help mitigate forgetting in both underparameterized and overparameterized
settings. To validate our theoretical analysis, we conducted simulation
experiments on both linear regression models and Deep Neural Networks (DNNs).
Results from these simulations substantiate our theoretical findings.

通过随机梯度下降（SGD）在线性回归模型中提供了一般性的理论分析，揭示了任务序列与算法参数之间错综复杂的关系，探讨了连续学习中的灾难性遗忘和算法参数选取对任务学习顺序的影响，通过模拟实验验证了理论分析的结果。