Objective: Clinical knowledge enriched transformer models (e.g.,
ClinicalBERT) have state-of-the-art results on clinical NLP (natural language
processing) tasks. One of the core limitations of these transformer models is
the substantial memory consumption due to their full self-attention mechanism,
which leads to the performance degradation in long clinical texts. To overcome
this, we propose to leverage long-sequence transformer models (e.g., Longformer
and BigBird), which extend the maximum input sequence length from 512 to 4096,
to enhance the ability to model long-term dependencies in long clinical texts.
Materials and Methods: Inspired by the success of long sequence transformer
models and the fact that clinical notes are mostly long, we introduce two
domain enriched language models, Clinical-Longformer and Clinical-BigBird,
which are pre-trained on a large-scale clinical corpus. We evaluate both
language models using 10 baseline tasks including named entity recognition,
question answering, natural language inference, and document classification
tasks.
Results: The results demonstrate that Clinical-Longformer and
Clinical-BigBird consistently and significantly outperform ClinicalBERT and
other short-sequence transformers in all 10 downstream tasks and achieve new
state-of-the-art results.
Discussion: Our pre-trained language models provide the bedrock for clinical
NLP using long texts. We have made our source code available at
this https URL, and the pre-trained models
available for public download at:
this https URL.
Conclusion: This study demonstrates that clinical knowledge enriched
long-sequence transformers are able to learn long-term dependencies in long
clinical text. Our methods can also inspire the development of other
domain-enriched long-sequence transformers.

本研究提出了两种经过预训练的基于长序列 Transformer 模型的临床语言模型，它们能够有效地处理临床文本中的长期依赖关系，相较于传统短序列的模型，在十项基准测试任务中均取得了最先进的成果；该研究还提供了源代码和预训练的模型。