The advancement of natural language processing (NLP) systems in healthcare
hinges on language model ability to interpret the intricate information
contained within clinical notes. This process often requires integrating
information from various time points in a patient's medical history. However,
most earlier clinical language models were pretrained with a context length
limited to roughly one clinical document. In this study, We introduce
ClinicalMamba, a specialized version of the Mamba language model, pretrained on
a vast corpus of longitudinal clinical notes to address the unique linguistic
characteristics and information processing needs of the medical domain.
ClinicalMamba, with 130 million and 2.8 billion parameters, demonstrates a
superior performance in modeling clinical language across extended text lengths
compared to Mamba and clinical Llama. With few-shot learning, ClinicalMamba
achieves notable benchmarks in speed and accuracy, outperforming existing
clinical language models and general domain large models like GPT-4 in
longitudinal clinical notes information extraction tasks.

在医学领域，为了处理长期病例的复杂信息，我们引入了经过纵向临床笔记巨量预训练的专用语言模型 ClinicalMamba，参数量为 2.8 亿，在建模扩展文本长度方面显示出比 Mamba 和临床 Llama 更优越的性能，且在纵向临床笔记信息提取任务中，通过少样本学习实现了快速而准确的成果，胜过了现有的临床语言模型和 GPT-4 等大规模通用领域模型。