Motivation: A perennial challenge for biomedical researchers and clinical
practitioners is to stay abreast with the rapid growth of publications and
medical notes. Natural language processing (NLP) has emerged as a promising
direction for taming information overload. In particular, large neural language
models facilitate transfer learning by pretraining on unlabeled text, as
exemplified by the successes of BERT models in various NLP applications.
However, fine-tuning such models for an end task remains challenging,
especially with small labeled datasets, which are common in biomedical NLP.
Results: We conduct a systematic study on fine-tuning stability in biomedical
NLP. We show that finetuning performance may be sensitive to pretraining
settings, especially in low-resource domains. Large models have potential to
attain better performance, but increasing model size also exacerbates
finetuning instability. We thus conduct a comprehensive exploration of
techniques for addressing fine-tuning instability. We show that these
techniques can substantially improve fine-tuning performance for lowresource
biomedical NLP applications. Specifically, freezing lower layers is helpful for
standard BERT-BASE models, while layerwise decay is more effective for
BERT-LARGE and ELECTRA models. For low-resource text similarity tasks such as
BIOSSES, reinitializing the top layer is the optimal strategy. Overall,
domainspecific vocabulary and pretraining facilitate more robust models for
fine-tuning. Based on these findings, we establish new state of the art on a
wide range of biomedical NLP applications.
Availability and implementation: To facilitate progress in biomedical NLP, we
release our state-of-the-art pretrained and fine-tuned models:
this https URL

本文系统研究了微生物 NLP 中 fine-tuning 的稳定性，提出了一系列技术来解决这个问题，并通过领域特定词汇和预训练来建立更健壮的模型。

为生物医学自然语言处理调整大型神经语言模型

Fine-Tuning Large Neural Language Models for Biomedical Natural Language  Processing

Fine-tuning pre-trained transformer-based language models such as BERT has
become a common practice dominating leaderboards across various NLP benchmarks.
Despite the strong empirical performance of fine-tuned models, fine-tuning is
an unstable process: training the same model with multiple random seeds can
result in a large variance of the task performance. Previous literature (Devlin
et al., 2019; Lee et al., 2020; Dodge et al., 2020) identified two potential
reasons for the observed instability: catastrophic forgetting and small size of
the fine-tuning datasets. In this paper, we show that both hypotheses fail to
explain the fine-tuning instability. We analyze BERT, RoBERTa, and ALBERT,
fine-tuned on commonly used datasets from the GLUE benchmark, and show that the
observed instability is caused by optimization difficulties that lead to
vanishing gradients. Additionally, we show that the remaining variance of the
downstream task performance can be attributed to differences in generalization
where fine-tuned models with the same training loss exhibit noticeably
different test performance. Based on our analysis, we present a simple but
strong baseline that makes fine-tuning BERT-based models significantly more
stable than the previously proposed approaches. Code to reproduce our results
is available online: this https URL

通过分析 BERT、RoBERTa 和 ALBERT 在 GLUE 基准测试集上的表现，本文表明，Fine-tuning 不稳定是由于优化困难导致梯度消失，我们提出了一个简单但强大的基线方法，使 BERT-based 模型的 fine-tuning 显著更稳定。