In Continual Learning settings, deep neural networks are prone to
Catastrophic Forgetting. Orthogonal Gradient Descent was proposed to tackle the
challenge. However, no theoretical guarantees have been proven yet. We present
a theoretical framework to study Continual Learning algorithms in the Neural
Tangent Kernel regime. This framework comprises closed form expression of the
model through tasks and proxies for Transfer Learning, generalisation and tasks
similarity. In this framework, we prove that OGD is robust to Catastrophic
Forgetting then derive the first generalisation bound for SGD and OGD for
Continual Learning. Finally, we study the limits of this framework in practice
for OGD and highlight the importance of the Neural Tangent Kernel variation for
Continual Learning with OGD.

为了处理深度神经网络在连续学习环境下的灾难性遗忘问题，该研究提出了正交梯度下降算法。在神经切向核区间的理论框架下，证明了该算法鲁棒性较高，并推导出了用于连续学习的 SGD 和 OGD 的首个推广限制。最后，研究了 OGD 的局限性，并强调了神经切向核变化对于 OGD 连续学习问题的重要性。