The advancement of natural language processing (NLP) systems in healthcare hinges on language model ability to interpret the intricate information contained within clinical notes. This process often requires integrating information from various time points in a patient's medical history. However, most earlier clinical language models were pretrained with a context length limited to roughly one clinical document. In this study, We introduce ClinicalMamba, a specialized version of the Mamba language model, pretrained on a vast corpus of longitudinal clinical notes to address the unique linguistic characteristics and information processing needs of the medical domain. ClinicalMamba, with 130 million and 2.8 billion parameters, demonstrates a superior performance in modeling clinical language across extended text lengths compared to Mamba and clinical Llama. With few-shot learning, ClinicalMamba achieves notable benchmarks in speed and accuracy, outperforming existing clinical language models and general domain large models like GPT-4 in longitudinal clinical notes information extraction tasks.

在医学领域，为了处理长期病例的复杂信息，我们引入了经过纵向临床笔记巨量预训练的专用语言模型ClinicalMamba，参数量为2.8亿，在建模扩展文本长度方面显示出比Mamba和临床Llama更优越的性能，且在纵向临床笔记信息提取任务中，通过少样本学习实现了快速而准确的成果，胜过了现有的临床语言模型和GPT-4等大规模通用领域模型。

临床小蟒: 横向临床记录的生成式临床语言模型