Survival analysis, as a challenging task, requires integrating Whole Slide
Images (WSIs) and genomic data for comprehensive decision-making. There are two
main challenges in this task: significant heterogeneity and complex inter- and
intra-modal interactions between the two modalities. Previous approaches
utilize co-attention methods, which fuse features from both modalities only
once after separate encoding. However, these approaches are insufficient for
modeling the complex task due to the heterogeneous nature between the
modalities. To address these issues, we propose a Biased Progressive Encoding
(BPE) paradigm, performing encoding and fusion simultaneously. This paradigm
uses one modality as a reference when encoding the other. It enables deep
fusion of the modalities through multiple alternating iterations, progressively
reducing the cross-modal disparities and facilitating complementary
interactions. Besides modality heterogeneity, survival analysis involves
various biomarkers from WSIs, genomics, and their combinations. The critical
biomarkers may exist in different modalities under individual variations,
necessitating flexible adaptation of the models to specific scenarios.
Therefore, we further propose a Mixture of Multimodal Experts (MoME) layer to
dynamically selects tailored experts in each stage of the BPE paradigm. Experts
incorporate reference information from another modality to varying degrees,
enabling a balanced or biased focus on different modalities during the encoding
process. Extensive experimental results demonstrate the superior performance of
our method on various datasets, including TCGA-BLCA, TCGA-UCEC and TCGA-LUAD.
Codes are available at this https URL

利用 Whole Slide Images 和基因组数据进行综合决策的生存分析面临着两个主要挑战：显著的异质性和两种模态之间的复杂相互作用。为解决这些问题，本文提出了一种偏向逐步编码的方法，通过多次迭代逐步减少跨模态差异并促进互补交互；此外，还提出了一种动态选择专家的多模态专家混合层来适应特定情况下的模型。广泛的实验结果表明我们的方法在各种数据集上表现出优越性能。