Survival risk stratification is an important step in clinical decision making
for breast cancer management. We propose a novel deep learning approach for
this purpose by integrating histopathological imaging, genetic and clinical
data. It employs vision transformers, specifically the MaxViT model, for image
feature extraction, and self-attention to capture intricate image relationships
at the patient level. A dual cross-attention mechanism fuses these features
with genetic data, while clinical data is incorporated at the final layer to
enhance predictive accuracy. Experiments on the public TCGA-BRCA dataset show
that our model, trained using the negative log likelihood loss function, can
achieve superior performance with a mean C-index of 0.64, surpassing existing
methods. This advancement facilitates tailored treatment strategies,
potentially leading to improved patient outcomes.

通过将组织病理图像、遗传数据和临床数据整合，构建了一种新颖的深度学习方法，使用视觉转换器和自注意力机制在病人层面上提取图像特征和捕获图像关系，同时采用双交叉注意力机制结合遗传数据，并在最终层级上加入临床数据以提高预测准确性，通过在公共 TCGA-BRCA 数据集上的实验，表明使用负对数似然损失函数训练的模型能够取得卓越的性能，均值 C 指数为 0.64，超过现有方法，为制定个体化的治疗策略提供了便利，潜在地改善了患者预后。

MM-SurvNet：基于深度学习的多模式数据融合在乳腺癌中的生存风险分层

MM-SurvNet: Deep Learning-Based Survival Risk Stratification in Breast  Cancer Through Multimodal Data Fusion

Breast cancer is a significant health concern affecting millions of women
worldwide. Accurate survival risk stratification plays a crucial role in
guiding personalised treatment decisions and improving patient outcomes. Here
we present BioFusionNet, a deep learning framework that fuses image-derived
features with genetic and clinical data to achieve a holistic patient profile
and perform survival risk stratification of ER+ breast cancer patients. We
employ multiple self-supervised feature extractors, namely DINO and MoCoV3,
pretrained on histopathology patches to capture detailed histopathological
image features. We then utilise a variational autoencoder (VAE) to fuse these
features, and harness the latent space of the VAE to feed into a self-attention
network, generating patient-level features. Next, we develop a
co-dual-cross-attention mechanism to combine the histopathological features
with genetic data, enabling the model to capture the interplay between them.
Additionally, clinical data is incorporated using a feed-forward network (FFN),
further enhancing predictive performance and achieving comprehensive multimodal
feature integration. Furthermore, we introduce a weighted Cox loss function,
specifically designed to handle imbalanced survival data, which is a common
challenge in the field. The proposed model achieves a mean concordance index
(C-index) of 0.77 and a time-dependent area under the curve (AUC) of 0.84,
outperforming state-of-the-art methods. It predicts risk (high versus low) with
prognostic significance for overall survival (OS) in univariate analysis
(HR=2.99, 95% CI: 1.88--4.78, p<0.005), and maintains independent significance
in multivariate analysis incorporating standard clinicopathological variables
(HR=2.91, 95% CI: 1.80--4.68, p<0.005). The proposed method not only improves
model performance but also addresses a critical gap in handling imbalanced
data.

利用深度学习和综合多模态特征集成方法，该研究提出了一种能够准确进行生存风险分层和乳腺癌患者个性化治疗决策的方法，并应用于 ER + 乳腺癌病人，取得了优于现有方法的预测性能，同时解决了处理不平衡数据的重要问题。