Histology analysis of the tumor micro-environment integrated with genomic
assays is the gold standard for most cancers in modern medicine. This paper
proposes a Gene-induced Multimodal Pre-training (GiMP) framework, which jointly
incorporates genomics and Whole Slide Images (WSIs) for classification tasks.
Our work aims at dealing with the main challenges of multi-modality image-omic
classification w.r.t. (1) the patient-level feature extraction difficulties
from gigapixel WSIs and tens of thousands of genes, and (2) effective fusion
considering high-order relevance modeling. Concretely, we first propose a group
multi-head self-attention gene encoder to capture global structured features in
gene expression cohorts. We design a masked patch modeling paradigm (MPM) to
capture the latent pathological characteristics of different tissues. The mask
strategy is randomly masking a fixed-length contiguous subsequence of patch
embeddings of a WSI. Finally, we combine the classification tokens of paired
modalities and propose a triplet learning module to learn high-order relevance
and discriminative patient-level information.After pre-training, a simple
fine-tuning can be adopted to obtain the classification results. Experimental
results on the TCGA dataset show the superiority of our network architectures
and our pre-training framework, achieving 99.47% in accuracy for image-omic
classification. The code is publicly available at
this https URL

该研究提出了一种基因诱导的多模态预训练（GiMP）框架，通过整合基因组学和全面切片图像（WSIs）进行分类任务，处理了多模态图像组合分类的主要挑战，并实现了 99.47% 的准确率。

基因诱导的多模态预训练用于图像组学分类

Gene-induced Multimodal Pre-training for Image-omic Classification

Deep learning has become the mainstream methodological choice for analyzing
and interpreting whole-slide digital pathology images (WSIs). It is commonly
assumed that tumor regions carry most predictive information. In this paper, we
proposed an unsupervised clustering-based multiple-instance learning, and apply
our method to develop deep-learning models for prediction of gene mutations
using WSIs from three cancer types in The Cancer Genome Atlas (TCGA) studies
(CRC, LUAD, and HNSCC). We showed that unsupervised clustering of image patches
could help identify predictive patches, exclude patches lack of predictive
information, and therefore improve prediction on gene mutations in all three
different cancer types, compared with the WSI based method without selection of
image patches and models based on only tumor regions. Additionally, our
proposed algorithm outperformed two recently published baseline algorithms
leveraging unsupervised clustering to assist model prediction. The
unsupervised-clustering-based approach for mutation prediction allows
identification of the spatial regions related to mutation of a specific gene
via the resolved probability scores, highlighting the heterogeneity of a
predicted genotype in the tumor microenvironment. Finally, our study also
demonstrated that selection of tumor regions of WSIs is not always the best way
to identify patches for prediction of gene mutations, and other tissue types in
the tumor micro-environment may provide better prediction ability for gene
mutations than tumor tissues.

本研究提出了一种基于非监督聚类的多实例学习方法，将其应用于三种癌症（CRC、LUAD 和 HNSCC）的 WSI 深度学习模型开发，使其能够更好地预测基因突变。结果显示，该方法可以识别预测性图像块，排除缺乏预测信息的图像块，并改善所有三种癌症类型中基因突变的预测效果。此外，我们的方法也能更好地识别预测基因突变的空间区域，证明 WSI 中的肿瘤区域不一定是最好的预测区域。