Multimodal survival methods combining gigapixel histology whole-slide images
(WSIs) and transcriptomic profiles are particularly promising for patient
prognostication and stratification. Current approaches involve tokenizing the
WSIs into smaller patches (>10,000 patches) and transcriptomics into gene
groups, which are then integrated using a Transformer for predicting outcomes.
However, this process generates many tokens, which leads to high memory
requirements for computing attention and complicates post-hoc interpretability
analyses. Instead, we hypothesize that we can: (1) effectively summarize the
morphological content of a WSI by condensing its constituting tokens using
morphological prototypes, achieving more than 300x compression; and (2)
accurately characterize cellular functions by encoding the transcriptomic
profile with biological pathway prototypes, all in an unsupervised fashion. The
resulting multimodal tokens are then processed by a fusion network, either with
a Transformer or an optimal transport cross-alignment, which now operates with
a small and fixed number of tokens without approximations. Extensive evaluation
on six cancer types shows that our framework outperforms state-of-the-art
methods with much less computation while unlocking new interpretability
analyses.

整合多模态存活方法、千亿像素组织学全幻灯片图像和转录组学，通过凝聚形态学原型和生物途径原型对 WSI 和转录组简化，再利用聚合网络（Transformer 或最优运输交叉对齐）处理得到的多模态令牌，能在较小且固定的令牌数量下实现高性能预测肿瘤患者预后、分层，同时实现较少的计算量并具备新的可解释性分析。