Histology analysis of the tumor micro-environment integrated with genomic
assays is the gold standard for most cancers in modern medicine. This paper
proposes a Gene-induced Multimodal Pre-training (GiMP) framework, which jointly
incorporates genomics and Whole Slide Images (WSIs) for classification tasks.
Our work aims at dealing with the main challenges of multi-modality image-omic
classification w.r.t. (1) the patient-level feature extraction difficulties
from gigapixel WSIs and tens of thousands of genes, and (2) effective fusion
considering high-order relevance modeling. Concretely, we first propose a group
multi-head self-attention gene encoder to capture global structured features in
gene expression cohorts. We design a masked patch modeling paradigm (MPM) to
capture the latent pathological characteristics of different tissues. The mask
strategy is randomly masking a fixed-length contiguous subsequence of patch
embeddings of a WSI. Finally, we combine the classification tokens of paired
modalities and propose a triplet learning module to learn high-order relevance
and discriminative patient-level information.After pre-training, a simple
fine-tuning can be adopted to obtain the classification results. Experimental
results on the TCGA dataset show the superiority of our network architectures
and our pre-training framework, achieving 99.47% in accuracy for image-omic
classification. The code is publicly available at
this https URL

该研究提出了一种基因诱导的多模态预训练（GiMP）框架，通过整合基因组学和全面切片图像（WSIs）进行分类任务，处理了多模态图像组合分类的主要挑战，并实现了 99.47% 的准确率。