Pre-trained transformer language models (LMs) have in recent years become the
dominant paradigm in applied NLP. These models have achieved state-of-the-art
performance on tasks such as information extraction, question answering,
sentiment analysis, document classification and many others. In the biomedical
domain, significant progress has been made in adapting this paradigm to NLP
tasks that require the integration of domain-specific knowledge as well as
statistical modelling of language. In particular, research in this area has
focused on the question of how best to construct LMs that take into account not
only the patterns of token distribution in medical text, but also the wealth of
structured information contained in terminology resources such as the UMLS.
This work contributes a data-centric paradigm for enriching the language
representations of biomedical transformer-encoder LMs by extracting text
sequences from the UMLS. This allows for graph-based learning objectives to be
combined with masked-language pre-training. Preliminary results from
experiments in the extension of pre-trained LMs as well as training from
scratch show that this framework improves downstream performance on multiple
biomedical and clinical Named Entity Recognition (NER) tasks.

通过从 UMLS 中提取文本序列，该工作为丰富生物医学变压器编码器的语言表示做出了数据为中心的范例贡献，从而将基于图的学习目标与掩码语言预训练相结合，初步实验结果表明该框架提高了多个生物医学和临床命名实体识别任务的下游性能。