Multi-modal learning plays a crucial role in cancer diagnosis and prognosis.
Current deep learning based multi-modal approaches are often limited by their
abilities to model the complex correlations between genomics and histology
data, addressing the intrinsic complexity of tumour ecosystem where both tumour
and microenvironment contribute to malignancy. We propose a biologically
interpretative and robust multi-modal learning framework to efficiently
integrate histology images and genomics by decomposing the feature subspace of
histology images and genomics, reflecting distinct tumour and microenvironment
features. To enhance cross-modal interactions, we design a knowledge-driven
subspace fusion scheme, consisting of a cross-modal deformable attention module
and a gene-guided consistency strategy. Additionally, in pursuit of dynamically
optimizing the subspace knowledge, we further propose a novel gradient
coordination learning strategy. Extensive experiments demonstrate the
effectiveness of the proposed method, outperforming state-of-the-art techniques
in three downstream tasks of glioma diagnosis, tumour grading, and survival
analysis. Our code is available at
this https URL

通过分解组织学图像和基因组学的特征子空间，反映不同的肿瘤和微环境特征，我们提出了一种生物解释性和稳健的多模态学习框架，以高效地整合组织学图像和基因组学，从而解决了肿瘤生态系统中肿瘤和微环境共同增加恶性的内在复杂性问题。