Pre-trained transformer models have demonstrated success across many natural
language processing (NLP) tasks. In applying these models to the clinical
domain, a prevailing assumption is that pre-training language models from
scratch on large-scale biomedical data results in substantial improvements. We
test this assumption with 4 pathology classification tasks on a corpus of 2907
prostate cancer pathology reports. We evaluate 5 transformer pre-trained models
that are the same size but differ in pre-training corpora. Specifically, we
analyze 3 categories of models: 1)General-domain: BERT and Turing Natural
Language Representation (TNLR) models, which use general corpora for
pre-training, 2)Mixed-domain: BioBERT which is obtained from BERT by including
PubMed abstracts in pre-training and Clinical BioBERT which additionally
includes MIMIC-III clinical notes and 3)Domain-specific: PubMedBERT which is
pre-trained from scratch on PubMed abstracts. We find the mixed-domain and
domain-specific models exhibit faster feature disambiguation during
fine-tuning. However, the domain-specific model, PubMedBERT, can overfit to
minority classes when presented with class imbalance, a common scenario in
pathology report data. At the same time, the mixed-domain models are more
resistant to overfitting. Our findings indicate that the use of general natural
language and domain-specific corpora in pre-training serve complementary
purposes for pathology report classification. The first enables resistance to
overfitting when fine-tuning on an imbalanced dataset while the second allows
for more accurate modelling of the fine-tuning domain. An expert evaluation is
also conducted to reveal common outlier modes of each model. Our results could
inform better fine-tuning practices in the clinical domain, to possibly
leverage the benefits of mixed-domain models for imbalanced downstream
datasets.

测试 4 个病理分类任务，涉及 2907 份前列腺癌病理报告，评估 5 个变压器预训练模型以及他们的不同预训练语料库，发现混合域和领域特定的模型在微调过程中具有更快的特征消歧能力，同时，混合域模型更具抗过度拟合性。同时，使用通用自然语言和特定领域语料库在预训练中互补起到了病理报告分类的作用。