Using language models (LMs) pre-trained in a self-supervised setting on large
corpora and then fine-tuning for a downstream task has helped to deal with the
problem of limited label data for supervised learning tasks such as Named
Entity Recognition (NER). Recent research in biomedical language processing has
offered a number of biomedical LMs pre-trained using different methods and
techniques that advance results on many BioNLP tasks, including NER. However,
there is still a lack of a comprehensive comparison of pre-training approaches
that would work more optimally in the biomedical domain. This paper aims to
investigate different pre-training methods, such as pre-training the biomedical
LM from scratch and pre-training it in a continued fashion. We compare existing
methods with our proposed pre-training method of initializing weights for new
tokens by distilling existing weights from the BERT model inside the context
where the tokens were found. The method helps to speed up the pre-training
stage and improve performance on NER. In addition, we compare how masking rate,
corruption strategy, and masking strategies impact the performance of the
biomedical LM. Finally, using the insights from our experiments, we introduce a
new biomedical LM (BIOptimus), which is pre-trained using Curriculum Learning
(CL) and contextualized weight distillation method. Our model sets new states
of the art on several biomedical Named Entity Recognition (NER) tasks. We
release our code and all pre-trained models

本研究通过比较不同的预训练方法，包括从头开始预训练生物医学语言模型和在连续环境中预训练模型，并利用 BERT 模型内上下文中的现有权重提炼初始化新标记的权重，加快预训练阶段并提高命名实体识别性能；此外，我们还比较了掩码率、损坏策略和掩码策略对生物医学语言模型性能的影响；最终，我们通过课程学习和上下文化权重提炼方法提出了一种新的生物医学语言模型 (BIOptimus)，在多项生物医学命名实体识别任务中创造了新的技术水平