Specialised pre-trained language models are becoming more frequent in NLP
since they can potentially outperform models trained on generic texts. BioBERT
and BioClinicalBERT are two examples of such models that have shown promise in
medical NLP tasks. Many of these models are overparametrised and
resource-intensive, but thanks to techniques like Knowledge Distillation (KD),
it is possible to create smaller versions that perform almost as well as their
larger counterparts. In this work, we specifically focus on development of
compact language models for processing clinical texts (i.e. progress notes,
discharge summaries etc). We developed a number of efficient lightweight
clinical transformers using knowledge distillation and continual learning, with
the number of parameters ranging from 15 million to 65 million. These models
performed comparably to larger models such as BioBERT and ClinicalBioBERT and
significantly outperformed other compact models trained on general or
biomedical data. Our extensive evaluation was done across several standard
datasets and covered a wide range of clinical text-mining tasks, including
Natural Language Inference, Relation Extraction, Named Entity Recognition, and
Sequence Classification. To our knowledge, this is the first comprehensive
study specifically focused on creating efficient and compact transformers for
clinical NLP tasks. The models and code used in this study can be found on our
Huggingface profile at this https URL and Github page at
this https URL,
respectively, promoting reproducibility of our results.

通过知识蒸馏和不断学习等技术，本研究着眼于发展处理临床文本的高效轻量级语言模型，与生物医学文本数据训练的大型模型相比，这些模型效果相当甚至更好，并超越了任何其他训练过的小型模型在自然语言推理、关系提取、命名实体识别和序列分类等临床文本挖掘任务上的表现。