The International Classification of Diseases (ICD) system is the
international standard for classifying diseases and procedures during a
healthcare encounter and is widely used for healthcare reporting and management
purposes. Assigning correct codes for clinical procedures is important for
clinical, operational, and financial decision-making in healthcare. Contextual
word embedding models have achieved state-of-the-art results in multiple NLP
tasks. However, these models have yet to achieve state-of-the-art results in
the ICD classification task since one of their main disadvantages is that they
can only process documents that contain a small number of tokens which is
rarely the case with real patient notes. In this paper, we introduce ICDBigBird
a BigBird-based model which can integrate a Graph Convolutional Network (GCN),
that takes advantage of the relations between ICD codes in order to create
'enriched' representations of their embeddings, with a BigBird contextual model
that can process larger documents. Our experiments on a real-world clinical
dataset demonstrate the effectiveness of our BigBird-based model on the ICD
classification task as it outperforms the previous state-of-the-art models.

本文介绍 ICDBigBird，一种基于 BigBird 模型的方法，可以将图卷积网络（GCN）与一个可以处理更大文档的 BigBird 上下文模型相结合，以创造出其嵌入的 'enriched' 描述，实验结果表明，ICDBigBird 在 ICD 分类任务上具有显著的优势。

ICDBigBird: 一种基于上下文嵌入模型的 ICD 码分类模型

ICDBigBird: A Contextual Embedding Model for ICD Code Classification

Contextual word embedding models, such as BioBERT and Bio_ClinicalBERT, have
achieved state-of-the-art results in biomedical natural language processing
tasks by focusing their pre-training process on domain-specific corpora.
However, such models do not take into consideration expert domain knowledge.
In this work, we introduced UmlsBERT, a contextual embedding model that
integrates domain knowledge during the pre-training process via a novel
knowledge augmentation strategy. More specifically, the augmentation on
UmlsBERT with the Unified Medical Language System (UMLS) Metathesaurus was
performed in two ways: i) connecting words that have the same underlying
`concept' in UMLS, and ii) leveraging semantic group knowledge in UMLS to
create clinically meaningful input embeddings. By applying these two
strategies, UmlsBERT can encode clinical domain knowledge into word embeddings
and outperform existing domain-specific models on common named-entity
recognition (NER) and clinical natural language inference clinical NLP tasks.

UmlsBERT is a contextual embedding model for biomedical natural language processing that integrates domain knowledge during pre-training via a novel knowledge augmentation strategy, outperforming existing models on named-entity recognition and clinical natural language inference tasks.

UmlsBERT: 使用统一医学语言系统词表增强上下文嵌入的临床领域知识

UmlsBERT: Clinical Domain Knowledge Augmentation of Contextual  Embeddings Using the Unified Medical Language System Metathesaurus

Contextual word embedding models such as ELMo (Peters et al., 2018) and BERT
(Devlin et al., 2018) have dramatically improved performance for many natural
language processing (NLP) tasks in recent months. However, these models have
been minimally explored on specialty corpora, such as clinical text; moreover,
in the clinical domain, no publicly-available pre-trained BERT models yet
exist. In this work, we address this need by exploring and releasing BERT
models for clinical text: one for generic clinical text and another for
discharge summaries specifically. We demonstrate that using a domain-specific
model yields performance improvements on three common clinical NLP tasks as
compared to nonspecific embeddings. These domain-specific models are not as
performant on two clinical de-identification tasks, and argue that this is a
natural consequence of the differences between de-identified source text and
synthetically non de-identified task text.

本文探讨了基于词汇的上下文嵌入模型在临床领域的应用，发现相对于通用语料库，专业领域语料库下的 BERT 模型在三个典型的临床自然语言处理任务上表现更加出色。