When machine learning models are trained continually on a sequence of tasks,
they are liable to forget what they learned on previous tasks -- a phenomenon
known as catastrophic forgetting. Proposed solutions to catastrophic forgetting
tend to involve storing information about past tasks, meaning that memory usage
is a chief consideration in determining their practicality. This paper proposes
a memory-efficient solution to catastrophic forgetting, improving upon an
established algorithm known as orthogonal gradient descent (OGD). OGD utilizes
prior model gradients to find weight updates that preserve performance on prior
datapoints. However, since the memory cost of storing prior model gradients
grows with the runtime of the algorithm, OGD is ill-suited to continual
learning over arbitrarily long time horizons. To address this problem, this
paper proposes SketchOGD. SketchOGD employs an online sketching algorithm to
compress model gradients as they are encountered into a matrix of a fixed,
user-determined size. In contrast to existing memory-efficient variants of OGD,
SketchOGD runs online without the need for advance knowledge of the total
number of tasks, is simple to implement, and is more amenable to analysis. We
provide theoretical guarantees on the approximation error of the relevant
sketches under a novel metric suited to the downstream task of OGD.
Experimentally, we find that SketchOGD tends to outperform current
state-of-the-art variants of OGD given a fixed memory budget.

本文提出 SketchOGD 作为解决连续任务中机器学习模型遗忘的内存高效算法，使用在线素描算法压缩模型梯度，并在理论和实验上进行了评估。

SketchOGD：内存高效的持续学习

SketchOGD: Memory-Efficient Continual Learning

In this report we consider the following problem: Given a trained model that
is partially faulty, can we correct its behaviour without having to train the
model from scratch? In other words, can we ``debug" neural networks similar to
how we address bugs in our mathematical models and standard computer code. We
base our approach on the hypothesis that debugging can be treated as a two-task
continual learning problem. In particular, we employ a modified version of a
continual learning algorithm called Orthogonal Gradient Descent (OGD) to
demonstrate, via two simple experiments on the MNIST dataset, that we can
in-fact \textit{unlearn} the undesirable behaviour while retaining the general
performance of the model, and we can additionally \textit{relearn} the
appropriate behaviour, both without having to train the model from scratch.

探讨了在不必重新训练模型的情况下，如何修复部分故障的神经网络，并使用改进的连续学习算法 OGD 来进行调试，实验证明可以实现消除不良行为和重新学习适当行为的目标。

使用正交梯度下降进行调试

Debugging using Orthogonal Gradient Descent

In Continual Learning settings, deep neural networks are prone to
Catastrophic Forgetting. Orthogonal Gradient Descent was proposed to tackle the
challenge. However, no theoretical guarantees have been proven yet. We present
a theoretical framework to study Continual Learning algorithms in the Neural
Tangent Kernel regime. This framework comprises closed form expression of the
model through tasks and proxies for Transfer Learning, generalisation and tasks
similarity. In this framework, we prove that OGD is robust to Catastrophic
Forgetting then derive the first generalisation bound for SGD and OGD for
Continual Learning. Finally, we study the limits of this framework in practice
for OGD and highlight the importance of the Neural Tangent Kernel variation for
Continual Learning with OGD.

为了处理深度神经网络在连续学习环境下的灾难性遗忘问题，该研究提出了正交梯度下降算法。在神经切向核区间的理论框架下，证明了该算法鲁棒性较高，并推导出了用于连续学习的 SGD 和 OGD 的首个推广限制。最后，研究了 OGD 的局限性，并强调了神经切向核变化对于 OGD 连续学习问题的重要性。