The integration of diverse clinical modalities such as medical imaging and
the tabular data obtained by the patients' Electronic Health Records (EHRs) is
a crucial aspect of modern healthcare. The integrative analysis of multiple
sources can provide a comprehensive understanding of a patient's condition and
can enhance diagnoses and treatment decisions. Deep Neural Networks (DNNs)
consistently showcase outstanding performance in a wide range of multimodal
tasks in the medical domain. However, the complex endeavor of effectively
merging medical imaging with clinical, demographic and genetic information
represented as numerical tabular data remains a highly active and ongoing
research pursuit.
We present a novel framework based on hypernetworks to fuse clinical imaging
and tabular data by conditioning the image processing on the EHR's values and
measurements. This approach aims to leverage the complementary information
present in these modalities to enhance the accuracy of various medical
applications. We demonstrate the strength and the generality of our method on
two different brain Magnetic Resonance Imaging (MRI) analysis tasks, namely,
brain age prediction conditioned by subject's sex, and multiclass Alzheimer's
Disease (AD) classification conditioned by tabular data. We show that our
framework outperforms both single-modality models and state-of-the-art
MRI-tabular data fusion methods. The code, enclosed to this manuscript will be
made publicly available.

基于超网络的医学成像与表格数据融合框架，通过将图像处理与电子病历数值和测量值相结合，旨在提高各种医学应用的准确性。

超级融合：一种基于超网络的表格和医学影像数据多模态融合预测建模方法

HyperFusion: A Hypernetwork Approach to Multimodal Integration of  Tabular and Medical Imaging Data for Predictive Modeling

Foundation large language models (LLMs) have shown an impressive ability to
solve tasks across a wide range of fields including health. To effectively
solve personalized health tasks, LLMs need the ability to ingest a diversity of
data modalities that are relevant to an individual's health status. In this
paper, we take a step towards creating multimodal LLMs for health that are
grounded in individual-specific data by developing a framework (HeLM: Health
Large Language Model for Multimodal Understanding) that enables LLMs to use
high-dimensional clinical modalities to estimate underlying disease risk. HeLM
encodes complex data modalities by learning an encoder that maps them into the
LLM's token embedding space and for simple modalities like tabular data by
serializing the data into text. Using data from the UK Biobank, we show that
HeLM can effectively use demographic and clinical features in addition to
high-dimensional time-series data to estimate disease risk. For example, HeLM
achieves an AUROC of 0.75 for asthma prediction when combining tabular and
spirogram data modalities compared with 0.49 when only using tabular data.
Overall, we find that HeLM outperforms or performs at parity with classical
machine learning approaches across a selection of eight binary traits.
Furthermore, we investigate the downstream uses of this model such as its
generalizability to out-of-distribution traits and its ability to power
conversations around individual health and wellness.

该研究通过开发一种名为 HeLM 的框架，使多模态大型语言模型 (LLMs) 能够使用高维临床模态来估计潜在疾病风险，并且在使用英国生物银行的数据进行验证时，HeLM 能够有效地利用人口统计学、临床特征和高维时间序列数据来估计疾病风险。