Pre-trained language models (PLMs) demonstrate excellent abilities to
understand texts in the generic domain while struggling in a specific domain.
Although continued pre-training on a large domain-specific corpus is effective,
it is costly to tune all the parameters on the domain. In this paper, we
investigate whether we can adapt PLMs both effectively and efficiently by only
tuning a few parameters. Specifically, we decouple the feed-forward networks
(FFNs) of the Transformer architecture into two parts: the original pre-trained
FFNs to maintain the old-domain knowledge and our novel domain-specific
adapters to inject domain-specific knowledge in parallel. Then we adopt a
mixture-of-adapters gate to fuse the knowledge from different domain adapters
dynamically. Our proposed Mixture-of-Domain-Adapters (MixDA) employs a
two-stage adapter-tuning strategy that leverages both unlabeled data and
labeled data to help the domain adaptation: i) domain-specific adapter on
unlabeled data; followed by ii) the task-specific adapter on labeled data.
MixDA can be seamlessly plugged into the pretraining-finetuning paradigm and
our experiments demonstrate that MixDA achieves superior performance on
in-domain tasks (GLUE), out-of-domain tasks (ChemProt, RCT, IMDB, Amazon), and
knowledge-intensive tasks (KILT). Further analyses demonstrate the reliability,
scalability, and efficiency of our method. The code is available at
this https URL

本文介绍了一种基于预训练语言模型和适应领域的新型领域自适应方法，通过两个阶段的适配器调整策略（在无标签数据上先进行领域适配器调整，然后进行标签数据上的任务适配器调整），可以同时提高在特定领域、跨领域和知识密集型任务中的性能。