Cancer clinics capture disease data at various scales, from genetic to organ
level. Current bioinformatic methods struggle to handle the heterogeneous
nature of this data, especially with missing modalities. We propose PARADIGM, a
Graph Neural Network (GNN) framework that learns from multimodal, heterogeneous
datasets to improve clinical outcome prediction. PARADIGM generates embeddings
from multi-resolution data using foundation models, aggregates them into
patient-level representations, fuses them into a unified graph, and enhances
performance for tasks like survival analysis. We train GNNs on pan-Squamous
Cell Carcinomas and validate our approach on Moffitt Cancer Center lung SCC
data. Multimodal GNN outperforms other models in patient survival prediction.
Converging individual data modalities across varying scales provides a more
insightful disease view. Our solution aims to understand the patient's
circumstances comprehensively, offering insights on heterogeneous data
integration and the benefits of converging maximum data views.

用于患者生存预测的基于多模态数据的图神经网络框架 PARADIGM 优于其他模型，通过整合多种数据视角，提供全面的疾病认知。

基于嵌入式多模式学习的全面鳞癌的生存结果改善

Embedding-based Multimodal Learning on Pan-Squamous Cell Carcinomas for  Improved Survival Outcomes

Cancer prognosis and survival outcome predictions are crucial for therapeutic
response estimation and for stratifying patients into various treatment groups.
Medical domains concerned with cancer prognosis are abundant with multiple
modalities, including pathological image data and non-image data such as
genomic information. To date, multimodal learning has shown potential to
enhance clinical prediction model performance by extracting and aggregating
information from different modalities of the same subject. This approach could
outperform single modality learning, thus improving computer-aided diagnosis
and prognosis in numerous medical applications. In this work, we propose a
cross-modality attention-based multimodal fusion pipeline designed to integrate
modality-specific knowledge for patient survival prediction in non-small cell
lung cancer (NSCLC). Instead of merely concatenating or summing up the features
from different modalities, our method gauges the importance of each modality
for feature fusion with cross-modality relationship when infusing the
multimodal features. Compared with single modality, which achieved c-index of
0.5772 and 0.5885 using solely tissue image data or RNA-seq data, respectively,
the proposed fusion approach achieved c-index 0.6587 in our experiment,
showcasing the capability of assimilating modality-specific knowledge from
varied modalities.

提出了一种跨模态注意力多模态融合管道，用于整合非小细胞肺癌（NSCLC）患者生存预测的特定模态知识，相较于单一模态的组合方法，在实验中取得了 c - 指数为 0.6587 的结果，展示了从不同模态融合中融入特定知识的能力。