Contextual word embedding models, such as BioBERT and Bio_ClinicalBERT, have
achieved state-of-the-art results in biomedical natural language processing
tasks by focusing their pre-training process on domain-specific corpora.
However, such models do not take into consideration expert domain knowledge.
In this work, we introduced UmlsBERT, a contextual embedding model that
integrates domain knowledge during the pre-training process via a novel
knowledge augmentation strategy. More specifically, the augmentation on
UmlsBERT with the Unified Medical Language System (UMLS) Metathesaurus was
performed in two ways: i) connecting words that have the same underlying
`concept' in UMLS, and ii) leveraging semantic group knowledge in UMLS to
create clinically meaningful input embeddings. By applying these two
strategies, UmlsBERT can encode clinical domain knowledge into word embeddings
and outperform existing domain-specific models on common named-entity
recognition (NER) and clinical natural language inference clinical NLP tasks.

UmlsBERT is a contextual embedding model for biomedical natural language processing that integrates domain knowledge during pre-training via a novel knowledge augmentation strategy, outperforming existing models on named-entity recognition and clinical natural language inference tasks.