Chronic diseases such as diabetes are the leading causes of morbidity and mortality worldwide. Numerous research studies have been attempted with various deep learning models in diagnosis. However, most previous studies had certain limitations, including using publicly available datasets (e.g. MIMIC), and imbalanced data. In this study, we collected five-year electronic health records (EHRs) from the Taiwan hospital database, including 1,420,596 clinical notes, 387,392 laboratory test results, and more than 1,505 laboratory test items, focusing on research pre-training large language models. We proposed a novel Large Language Multimodal Models (LLMMs) framework incorporating multimodal data from clinical notes and laboratory test results for the prediction of chronic disease risk. Our method combined a text embedding encoder and multi-head attention layer to learn laboratory test values, utilizing a deep neural network (DNN) module to merge blood features with chronic disease semantics into a latent space. In our experiments, we observe that clinicalBERT and PubMed-BERT, when combined with attention fusion, can achieve an accuracy of 73% in multiclass chronic diseases and diabetes prediction. By transforming laboratory test values into textual descriptions and employing the Flan T-5 model, we achieved a 76% Area Under the ROC Curve (AUROC), demonstrating the effectiveness of leveraging numerical text data for training and inference in language models. This approach significantly improves the accuracy of early-stage diabetes prediction.

通过收集来自台湾医院数据库的五年电子健康记录，该研究提出了一个新颖的大型语言多模型（LLMMs）框架，将临床笔记和实验室检测结果的多模态数据结合起来预测慢性疾病风险，观察到将clinicalBERT和PubMed-BERT与注意力融合相结合可以在多类慢性疾病和糖尿病预测中达到73%的准确率，将实验室检测值转化为文本描述后，利用Flan T-5模型可以达到76%的ROC曲线下面积（AUROC），从而显著提高了糖尿病早期预测的准确性。

利用电子病历数据进行5年慢性病队列预测的大型多模态语言模型