预训练语言模型的廉价领域自适应: 生物医学实体识别和 Covid-19 问答的案例研究

Apr, 2020

预训练语言模型的廉价领域自适应: 生物医学实体识别和 Covid-19 问答的案例研究

Inexpensive Domain Adaptation of Pretrained Language Models: A Case Study on Biomedical Named Entity Recognition

Nina Poerner, Ulli Waltinger, Hinrich Schütze

TL;DR本文介绍了基于目标域文本的Word2Vec的训练和对齐策略，用于适应已有的预训练语言模型，并在生物医学实体识别任务和Covid-19问答任务上取得了60%的BioBERT-BERT F1差距的覆盖率，且成本仅相当于BioBERT的5%的CO2排放和2%的云计算成本。

Abstract

domain adaptation of pretrained language models (PTLMs) is typically achieved by pretraining on in-domain text. While successful, this approach is expensive in terms of hardware, runtime and CO_2 emissions. Here,