Continual Learning (CL) aims to learn a sequence of problems (i.e., tasks and
domains) by transferring knowledge acquired on previous problems, whilst
avoiding forgetting of past ones. Different from previous approaches which
focused on CL for one NLP task or domain in a specific use-case, in this paper,
we address a more general CL setting to learn from a sequence of problems in a
unique framework. Our method, HOP, permits to hop across tasks and domains by
addressing the CL problem along three directions: (i) we employ a set of
adapters to generalize a large pre-trained model to unseen problems, (ii) we
compute high-order moments over the distribution of embedded representations to
distinguish independent and correlated statistics across different tasks and
domains, (iii) we process this enriched information with auxiliary heads
specialized for each end problem. Extensive experimental campaign on 4 NLP
applications, 5 benchmarks and 2 CL setups demonstrates the effectiveness of
our HOP.

HOP 方法使用适配器泛化预训练模型、计算嵌入表示分布的高阶矩来区分不同任务和领域的独立和相关统计信息，并使用针对每个最终问题专门设计的辅助头处理丰富的信息，证明了其在 4 个 NLP 应用，5 个基准测试和 2 个 Continual Learning 设置中的有效性。