Cancer diagnosis, prognosis and therapeutic response predictions are based on morphological information from histology slides and molecular profiles from genomic data. However, most deep learning-based objective outcome prediction and grading paradigms are based on histology or genomics alone and do not make use of the complimentary information in an intuitive manner. In this work, we propose Pathomic Fusion, a strategy for end-to-end multimodal fusion of histology image and genomic (mutations, CNV, mRNAseq) features for survival outcome prediction and patient stratification, where histologic features are also learned via graph convolutional networks (GCNs). Our approach models pairwise feature interactions across modalities by taking the Kronecker product of gated feature representations, and controls the expressiveness of each representation via a gating-based attention mechanism. The proposed framework is able to model pairwise interactions across features in different modalities and control their relative importance. We validate our approach using glioma datasets from the Cancer Genome Atlas (TCGA), which contains paired whole-slide image, genotype, and transcriptome data with ground truth survival and histologic grade labels. Based on a rigorous 15-fold cross validation, our results demonstrate that the proposed multimodal fusion paradigm improves prognostic determinations from grading and molecular subtyping as well as unimodal deep networks trained on histology and genomic data alone. The proposed method establishes insight and theory on how to train deep networks on multimodal biomedical data in an intuitive manner, which will be useful for other problems in medicine that seek to combine heterogeneous data streams for understanding diseases and predicting response and resistance to treatment.

本文提出了一种解释性的、端到端的多模态融合策略—— Pathomic Fusion，它基于组织学图像和基因组学（突变、CNV、RNA-Seq）特征，模型化跨模式的特征交互，并通过基于门控的注意机制控制每个表示的表现力，以提高生存结果预测的准确性，从而有助于其他医学问题的解决。

Pathomic Fusion：融合组织学和基因组特征的癌症诊断和预后一体化框架