The rapidly emerging field of deep learning-based computational pathology has
shown promising results in utilizing whole slide images (WSIs) to objectively
prognosticate cancer patients. However, most prognostic methods are currently
limited to either histopathology or genomics alone, which inevitably reduces
their potential to accurately predict patient prognosis. Whereas integrating
WSIs and genomic features presents three main challenges: (1) the enormous
heterogeneity of gigapixel WSIs which can reach sizes as large as
150,000x150,000 pixels; (2) the absence of a spatially corresponding
relationship between histopathology images and genomic molecular data; and (3)
the existing early, late, and intermediate multimodal feature fusion strategies
struggle to capture the explicit interactions between WSIs and genomics. To
ameliorate these issues, we propose the Mutual-Guided Cross-Modality
Transformer (MGCT), a weakly-supervised, attention-based multimodal learning
framework that can combine histology features and genomic features to model the
genotype-phenotype interactions within the tumor microenvironment. To validate
the effectiveness of MGCT, we conduct experiments using nearly 3,600 gigapixel
WSIs across five different cancer types sourced from The Cancer Genome Atlas
(TCGA). Extensive experimental results consistently emphasize that MGCT
outperforms the state-of-the-art (SOTA) methods.

基于深度学习的计算病理学领域的研究表明，利用全切片图像（WSIs）客观预测癌症患者预后的效果显著。然而，目前大多数预后方法局限于组织病理学或基因组学，不可避免地降低了其准确预测患者预后的潜力。为了解决这些问题，我们提出了一种弱监督、基于注意机制的多模态学习框架 —— 互相引导的跨模态转换器（MGCT），可以利用组织学特征和基因组特征来建模肿瘤微环境内的基因型 - 表型相互作用。通过使用来自癌症基因组图谱（TCGA）的五种不同癌症类型的近 3600 个 Gigapixel WSIs 进行实验，广泛的实验结果一致表明 MGCT 优于现有的最先进方法（SOTA）。

MGCT: 基于综合组织病理学 - 基因组特征的生存结局预测的互相引导的跨模态变换器

MGCT: Mutual-Guided Cross-Modality Transformer for Survival Outcome  Prediction using Integrative Histopathology-Genomic Features

Most prognostic methods require a decent amount of data for model training.
In reality, however, the amount of historical data owned by a single
organization might be small or not large enough to train a reliable prognostic
model. To address this challenge, this article proposes a federated prognostic
model that allows multiple users to jointly construct a failure time prediction
model using their multi-stream, high-dimensional, and incomplete data while
keeping each user's data local and confidential. The prognostic model first
employs multivariate functional principal component analysis to fuse the
multi-stream degradation signals. Then, the fused features coupled with the
times-to-failure are utilized to build a (log)-location-scale regression model
for failure prediction. To estimate parameters using distributed datasets and
keep the data privacy of all participants, we propose a new federated algorithm
for feature extraction. Numerical studies indicate that the performance of the
proposed model is the same as that of classic non-federated prognostic models
and is better than that of the models constructed by each user itself.

本文提出了一种联邦预测模型，允许多个用户共同构建基于多流、高维和不完整数据的故障时间预测模型，同时保持每个用户的数据本地和保密。该预测模型首先使用多元功能主成分分析融合多流退化信号，然后利用融合特征和失效时间来构建（对数）位置 - 比例回归模型进行故障预测。为了使用分布式数据集进行参数估计并保护所有参与者的数据隐私，我们提出了一种新的联邦算法进行特征提取。数值研究表明，所提出的模型的性能与经典的非联邦预测模型相同，并且优于每个用户自建模型的性能。