Most natural language tasks in the radiology domain use language models pre-trained on biomedical corpus. There are few pretrained language models trained specifically for radiology, and fewer still that have been trained in a low data setting and gone on to produce comparable results in fine-tuning tasks. We present RadLing, a continuously pretrained language model using Electra-small (Clark et al., 2020) architecture, trained using over 500K radiology reports, that can compete with state-of-the-art results for fine tuning tasks in radiology domain. Our main contribution in this paper is knowledge-aware masking which is a taxonomic knowledge-assisted pretraining task that dynamically masks tokens to inject knowledge during pretraining. In addition, we also introduce an knowledge base-aided vocabulary extension to adapt the general tokenization vocabulary to radiology domain.

本文介绍了RadLing这一预训练模型，它使用了500K个放射学报告作为训练集，并针对放射学领域进行了微调，使其在细化调整任务中取得了与最先进结果相当的性能。本文的主要贡献在于知识感知遮蔽，这是一种通过遮蔽单词来注入知识的分类知识支持的预训练任务。此外，本文还介绍了一种知识库辅助的词汇扩展方法，以适应放射学领域的通用分词词汇。

RadLing：面向高效放射学报告理解