Pre-trained transformer language models (LMs) have in recent years become the
dominant paradigm in applied NLP. These models have achieved state-of-the-art
performance on tasks such as information extraction, question answering,
sentiment analysis, document classification and many others. In the biomedical
domain, significant progress has been made in adapting this paradigm to NLP
tasks that require the integration of domain-specific knowledge as well as
statistical modelling of language. In particular, research in this area has
focused on the question of how best to construct LMs that take into account not
only the patterns of token distribution in medical text, but also the wealth of
structured information contained in terminology resources such as the UMLS.
This work contributes a data-centric paradigm for enriching the language
representations of biomedical transformer-encoder LMs by extracting text
sequences from the UMLS. This allows for graph-based learning objectives to be
combined with masked-language pre-training. Preliminary results from
experiments in the extension of pre-trained LMs as well as training from
scratch show that this framework improves downstream performance on multiple
biomedical and clinical Named Entity Recognition (NER) tasks.

通过从 UMLS 中提取文本序列，该工作为丰富生物医学变压器编码器的语言表示做出了数据为中心的范例贡献，从而将基于图的学习目标与掩码语言预训练相结合，初步实验结果表明该框架提高了多个生物医学和临床命名实体识别任务的下游性能。

UMLS-KGI-BERT: 基于数据中心的转换器中的生物医学实体识别知识集成

UMLS-KGI-BERT: Data-Centric Knowledge Integration in Transformers for  Biomedical Entity Recognition

Pre-trained transformer language models have shown remarkable performance on
a variety of NLP tasks. However, recent research has suggested that
phrase-level representations in these models reflect heavy influences of
lexical content, but lack evidence of sophisticated, compositional phrase
information. Here we investigate the impact of fine-tuning on the capacity of
contextualized embeddings to capture phrase meaning information beyond lexical
content. Specifically, we fine-tune models on an adversarial paraphrase
classification task with high lexical overlap, and on a sentiment
classification task. After fine-tuning, we analyze phrasal representations in
controlled settings following prior work. We find that fine-tuning largely
fails to benefit compositionality in these representations, though training on
sentiment yields a small, localized benefit for certain models. In follow-up
analyses, we identify confounding cues in the paraphrase dataset that may
explain the lack of composition benefits from that task, and we discuss
potential factors underlying the localized benefits from sentiment training.

研究发现，预训练的转换语言模型在很多 NLP 任务上表现出色；然而，这些模型在短语级别的表示中，除了词汇内容外，缺乏复杂的组合短语信息，进一步的 fine-tuning 只能在情感任务中局部提高性能，而在重新释义任务中则由于数据集中可能存在干扰信号的原因而不能提供改进。