In this work, we aim to establish a strong connection between two significant
bodies of machine learning research: continual learning and sequence modeling.
That is, we propose to formulate continual learning as a sequence modeling
problem, allowing advanced sequence models to be utilized for continual
learning. Under this formulation, the continual learning process becomes the
forward pass of a sequence model. By adopting the meta-continual learning (MCL)
framework, we can train the sequence model at the meta-level, on multiple
continual learning episodes. As a specific example of our new formulation, we
demonstrate the application of Transformers and their efficient variants as MCL
methods. Our experiments on seven benchmarks, covering both classification and
regression, show that sequence models can be an attractive solution for general
MCL.

我们在这项工作中旨在建立起机器学习研究的两个重要领域之间的紧密联系：连续学习和序列建模。也就是说，我们提出将连续学习作为序列建模问题进行建模，使得先进的序列模型可以用于连续学习。通过采用元连续学习（MCL）框架，我们可以在元级别上对序列模型进行训练，应用于多个连续学习实验。作为我们新建模的具体示例，我们演示了将 Transformers 及其高效变体用作 MCL 方法的应用。我们在七个基准测试上进行了实验，涵盖了分类和回归问题，结果表明序列模型可以成为通用连续学习的一个有吸引力的解决方案。

将持续学习重构为序列建模

Recasting Continual Learning as Sequence Modeling

Continual learning studies agents that learn from streams of tasks without
forgetting previous ones while adapting to new ones. Two recent
continual-learning scenarios have opened new avenues of research. In
meta-continual learning, the model is pre-trained to minimize catastrophic
forgetting of previous tasks. In continual-meta learning, the aim is to train
agents for faster remembering of previous tasks through adaptation. In their
original formulations, both methods have limitations. We stand on their
shoulders to propose a more general scenario, OSAKA, where an agent must
quickly solve new (out-of-distribution) tasks, while also requiring fast
remembering. We show that current continual learning, meta-learning,
meta-continual learning, and continual-meta learning techniques fail in this
new scenario. We propose Continual-MAML, an online extension of the popular
MAML algorithm as a strong baseline for this scenario. We empirically show that
Continual-MAML is better suited to the new scenario than the aforementioned
methodologies, as well as standard continual learning and meta-learning
approaches.

在研究中，提出了一个更普遍的场景 OSAKA，即代理人必须快速解决新的（分布外）任务，并快速回忆起以前的任务。作者提出了 Continual-MAML 作为强有力的基准线，并在实验中证明，它比其他方法更适合新情况。