Recently, domain-specific PLMs have been proposed to boost the task
performance of specific domains (e.g., biomedical and computer science) by
continuing to pre-train general PLMs with domain-specific corpora. However,
this Domain-Adaptive Pre-Training (DAPT; Gururangan et al. (2020)) tends to
forget the previous general knowledge acquired by general PLMs, which leads to
a catastrophic forgetting phenomenon and sub-optimal performance. To alleviate
this problem, we propose a new framework of General Memory Augmented
Pre-trained Language Model (G-MAP), which augments the domain-specific PLM by a
memory representation built from the frozen general PLM without losing any
general knowledge. Specifically, we propose a new memory-augmented layer, and
based on it, different augmented strategies are explored to build the memory
representation and then adaptively fuse it into the domain-specific PLM. We
demonstrate the effectiveness of G-MAP on various domains (biomedical and
computer science publications, news, and reviews) and different kinds (text
classification, QA, NER) of tasks, and the extensive results show that the
proposed G-MAP can achieve SOTA results on all tasks.

提出了一种 General Memory Augmented Pre-trained Language Model (G-MAP) 框架，通过从未失去通用知识的冻结通用 PLM 构建内存表示来增广专业领域 PLM，并探索不同的增广策略来将其自适应地融合到专业领域 PLM 中，证明了 G-MAP 对各种类型的任务，在各种领域中可以实现 SOTA 结果。