The rapidly emerging field of deep learning-based computational pathology has
shown promising results in utilizing whole slide images (WSIs) to objectively
prognosticate cancer patients. However, most prognostic methods are currently
limited to either histopathology or genomics alone, which inevitably reduces
their potential to accurately predict patient prognosis. Whereas integrating
WSIs and genomic features presents three main challenges: (1) the enormous
heterogeneity of gigapixel WSIs which can reach sizes as large as
150,000x150,000 pixels; (2) the absence of a spatially corresponding
relationship between histopathology images and genomic molecular data; and (3)
the existing early, late, and intermediate multimodal feature fusion strategies
struggle to capture the explicit interactions between WSIs and genomics. To
ameliorate these issues, we propose the Mutual-Guided Cross-Modality
Transformer (MGCT), a weakly-supervised, attention-based multimodal learning
framework that can combine histology features and genomic features to model the
genotype-phenotype interactions within the tumor microenvironment. To validate
the effectiveness of MGCT, we conduct experiments using nearly 3,600 gigapixel
WSIs across five different cancer types sourced from The Cancer Genome Atlas
(TCGA). Extensive experimental results consistently emphasize that MGCT
outperforms the state-of-the-art (SOTA) methods.

基于深度学习的计算病理学领域的研究表明，利用全切片图像（WSIs）客观预测癌症患者预后的效果显著。然而，目前大多数预后方法局限于组织病理学或基因组学，不可避免地降低了其准确预测患者预后的潜力。为了解决这些问题，我们提出了一种弱监督、基于注意机制的多模态学习框架 —— 互相引导的跨模态转换器（MGCT），可以利用组织学特征和基因组特征来建模肿瘤微环境内的基因型 - 表型相互作用。通过使用来自癌症基因组图谱（TCGA）的五种不同癌症类型的近 3600 个 Gigapixel WSIs 进行实验，广泛的实验结果一致表明 MGCT 优于现有的最先进方法（SOTA）。