Incremental Learning (IL) has been a long-standing problem in both vision and
Natural Language Processing (NLP) communities. In recent years, as Pre-trained
Language Models (PLMs) have achieved remarkable progress in various NLP
downstream tasks, utilizing PLMs as backbones has become a common practice in
recent research of IL in NLP. Most assume that catastrophic forgetting is the
biggest obstacle to achieving superior IL performance and propose various
techniques to overcome this issue. However, we find that this assumption is
problematic. Specifically, we revisit more than 20 methods on four
classification tasks (Text Classification, Intent Classification, Relation
Extraction, and Named Entity Recognition) under the two most popular IL
settings (Class-Incremental and Task-Incremental) and reveal that most of them
severely underestimate the inherent anti-forgetting ability of PLMs. Based on
the observation, we propose a frustratingly easy method called SEQ* for IL with
PLMs. The results show that SEQ* has competitive or superior performance
compared to state-of-the-art (SOTA) IL methods and requires considerably less
trainable parameters and training time. These findings urge us to revisit the
IL with PLMs and encourage future studies to have a fundamental understanding
of the catastrophic forgetting in PLMs. The data, code and scripts are publicly
available at
this https URL

在自然语言处理领域中，基于预训练语言模型的增量学习的研究表明，现有的方法普遍低估了预训练语言模型天生的抗遗忘能力，研究者提出了一种名为 SEQ * 的简单方法，它在各项度量上与最先进的增量学习方法具有竞争力或更好的表现，并且需要较少的可训练参数和训练时间。