Language modeling have shown impressive progress in generating compelling
text with good accuracy and high semantic coherence. An interesting research
direction is to augment these powerful models for specific applications using
contextual information. In this work, we explore multi-modal language modeling
for healthcare applications. We are interested in outcome prediction and
patient triage in hospital emergency department based on text information in
chief complaints and vital signs recorded at triage. We adapt Perceiver - a
modality-agnostic transformer-based model that has shown promising results in
several applications. Since vital-sign modality is represented in tabular
format, we modified Perceiver position encoding to ensure permutation
invariance. We evaluated the multi-modal language model for the task of
diagnosis code prediction using MIMIC-IV ED dataset on 120K visits. In the
experimental analysis, we show that mutli-modality improves the prediction
performance compared with models trained solely on text or vital signs. We
identified disease categories for which multi-modality leads to performance
improvement and show that for these categories, vital signs have added
predictive power. By analyzing the cross-attention layer, we show how
multi-modality contributes to model predictions. This work gives interesting
insights on the development of multi-modal language models for healthcare
applications.

本文探讨了多模式语言建模在医疗保健领域的应用，特别是基于急诊科主诉和生命体征文本信息的病人问题和需求预测，通过对 MIMIC-IV ED 数据集的诊断代码预测实验分析，我们发现多模式提高了预测性能，并且对一些疾病类别来说，生命体征具有更强的预测能力，并透过交叉注意层的分析，展示了多模式如何提升模型预测。