For predicting cancer survival outcomes, standard approaches in clinical
research are often based on two main modalities: pathology images for observing
cell morphology features, and genomic (e.g., bulk RNA-seq) for quantifying gene
expressions. However, existing pathology-genomic multi-modal algorithms face
significant challenges: (1) Valuable biological insights regarding genes and
gene-gene interactions are frequently overlooked; (2) one modality often
dominates the optimization process, causing inadequate training for the other
modality. In this paper, we introduce a new multi-modal ``Path-GPTOmic"
framework for cancer survival outcome prediction. First, to extract valuable
biological insights, we regulate the embedding space of a foundation model,
scGPT, initially trained on single-cell RNA-seq data, making it adaptable for
bulk RNA-seq data. Second, to address the imbalance-between-modalities problem,
we propose a gradient modulation mechanism tailored to the Cox partial
likelihood loss for survival prediction. The contributions of the modalities
are dynamically monitored and adjusted during the training process, encouraging
that both modalities are sufficiently trained. Evaluated on two TCGA(The Cancer
Genome Atlas) datasets, our model achieves substantially improved survival
prediction accuracy.

通过多模态框架和生物学洞察来提高癌症生存预测准确性。