Several biomedical language models have already been developed for clinical
language inference. However, these models typically utilize general
vocabularies and are trained on relatively small clinical corpora. We sought to
evaluate the impact of using a domain-specific vocabulary and a large clinical
training corpus on the performance of these language models in clinical
language inference. We trained a Bidirectional Encoder Decoder from
Transformers (BERT) model using a diverse, deidentified corpus of 75 million
deidentified clinical notes authored at the University of California, San
Francisco (UCSF). We evaluated this model on several clinical language
inference benchmark tasks: clinical and temporal concept recognition, relation
extraction and medical language inference. We also evaluated our model on two
tasks using discharge summaries from UCSF: diagnostic code assignment and
therapeutic class inference. Our model performs at par with the best publicly
available biomedical language models of comparable sizes on the public
benchmark tasks, and is significantly better than these models in a
within-system evaluation on the two tasks using UCSF data. The use of in-domain
vocabulary appears to improve the encoding of longer documents. The use of
large clinical corpora appears to enhance document encoding and inferential
accuracy. However, further research is needed to improve abbreviation
resolution, and numerical, temporal, and implicitly causal inference.

本文通过使用领域专有词汇和大规模临床训练语料库，使用双向编码器解码器（BERT）模型对医疗语言推理任务进行分析和评估，与公共基准任务上同等规模的公开可用的最佳生物医学语言模型相比，我们的模型表现相同，并且在使用 UCSF 数据的两个任务的系统内评估中优于这些模型，但需要进一步的研究来提高缩写、数字、时间和隐含因果推理的准确性。