Analyzing the genomic information from the Pan-Cancer database can help us
understand cancer-related factors and contribute to the cancer diagnosis and
prognosis. However, existing computational methods and deep learning methods
can not effectively find the deep correlations between tens of thousands of
genes, which leads to precision loss. In this paper, we proposed a novel
pretrained model called Gene-MOE to learn the general feature representations
of the Pan-Cancer dataset and transfer the pretrained weights to the downstream
tasks. The Gene-MOE fully exploits the mixture of expert (MOE) layers to learn
rich feature representations of high-dimensional genes. At the same time, we
build a mixture of attention expert (MOAE) model to learn the deep semantic
relationships within genetic features. Finally, we proposed a new
self-supervised pretraining strategy including loss function design, data
enhancement, and optimization strategy to train the Gene-MOE and further
improve the performance for the downstream analysis. We carried out cancer
classification and survival analysis experiments based on the Gene-MOE.
According to the survival analysis results on 14 cancer types, using Gene-MOE
outperformed state-of-the-art models on 12 cancer types. According to the
classification results, the total accuracy of the classification model for 33
cancer classifications reached 95.2\%. Through detailed feature analysis, we
found the Gene-MOE model can learn rich feature representations of
high-dimensional genes.

本文提出了一种名为 Gene-MOE 的新型预训练模型，通过深度学习方法在 Pan-Cancer 数据库中分析基因组信息，以了解与癌症相关的因素，并有助于癌症的诊断和预后。实验结果表明，Gene-MOE 模型在癌症分类和生存分析中表现优于现有的模型，具有学习高维基因的丰富特征表示的能力。