Survival prediction based on whole slide images (WSIs) is a challenging task
for patient-level multiple instance learning (MIL). Due to the vast amount of
data for a patient (one or multiple gigapixels WSIs) and the irregularly shaped
property of WSI, it is difficult to fully explore spatial, contextual, and
hierarchical interaction in the patient-level bag. Many studies adopt random
sampling pre-processing strategy and WSI-level aggregation models, which
inevitably lose critical prognostic information in the patient-level bag. In
this work, we propose a hierarchical vision Transformer framework named
HVTSurv, which can encode the local-level relative spatial information,
strengthen WSI-level context-aware communication, and establish patient-level
hierarchical interaction. Firstly, we design a feature pre-processing strategy,
including feature rearrangement and random window masking. Then, we devise
three layers to progressively obtain patient-level representation, including a
local-level interaction layer adopting Manhattan distance, a WSI-level
interaction layer employing spatial shuffle, and a patient-level interaction
layer using attention pooling. Moreover, the design of hierarchical network
helps the model become more computationally efficient. Finally, we validate
HVTSurv with 3,104 patients and 3,752 WSIs across 6 cancer types from The
Cancer Genome Atlas (TCGA). The average C-Index is 2.50-11.30% higher than all
the prior weakly supervised methods over 6 TCGA datasets. Ablation study and
attention visualization further verify the superiority of the proposed HVTSurv.
Implementation is available at: this https URL

提出了一种名为 HVTSurv 的分层视觉 Transformer 框架，可对全切片图像进行预测和患者级别的癌症预测，并通过 3,104 名患者和 3,752 个全切片图像的验证实现了比以前的弱监督方法更高的平均 C 指数。

HVTSurv: 基于整张切片图像的患者生存预测的分层视觉 Transformer 模型

HVTSurv: Hierarchical Vision Transformer for Patient-Level Survival  Prediction from Whole Slide Image

urvival prediction for cancer patients is critical for optimal treatment
selection and patient management. Current patient survival prediction methods
typically extract survival information from patients' clinical record data or
biological and imaging data. In practice, experienced clinicians can have a
preliminary assessment of patients' health status based on patients' observable
physical appearances, which are mainly facial features. However, such
assessment is highly subjective. In this work, the efficacy of objectively
capturing and using prognostic information contained in conventional portrait
photographs using deep learning for survival predication purposes is
investigated for the first time. A pre-trained StyleGAN2 model is fine-tuned on
a custom dataset of our cancer patients' photos to empower its generator with
generative ability suitable for patients' photos. The StyleGAN2 is then used to
embed the photographs to its highly expressive latent space. Utilizing the
state-of-the-art survival analysis models and based on StyleGAN's latent space
photo embeddings, this approach achieved a C-index of 0.677, which is notably
higher than chance and evidencing the prognostic value embedded in simple 2D
facial images. In addition, thanks to StyleGAN's interpretable latent space,
our survival prediction model can be validated for relying on essential facial
features, eliminating any biases from extraneous information like clothing or
background. Moreover, a health attribute is obtained from regression
coefficients, which has important potential value for patient care.

本文提出使用深度学习的方式对患者照片中抓取包含在传统人像照片中患者健康预测信息，并将其应用于癌症患者的生存预测，使用 StyleGAN2 取得了不错的效果，能够基于面部特征获得幸存率预测信息。

基于 StyleGAN 嵌入的人像照片深度学习预测癌症预后

Deep Learning for Cancer Prognosis Prediction Using Portrait Photos by  StyleGAN Embedding

Prognostic information at diagnosis has important implications for cancer
treatment and monitoring. Although cancer staging, histopathological
assessment, molecular features, and clinical variables can provide useful
prognostic insights, improving risk stratification remains an active research
area. We developed a deep learning system (DLS) to predict disease specific
survival across 10 cancer types from The Cancer Genome Atlas (TCGA). We used a
weakly-supervised approach without pixel-level annotations, and tested three
different survival loss functions. The DLS was developed using 9,086 slides
from 3,664 cases and evaluated using 3,009 slides from 1,216 cases. In
multivariable Cox regression analysis of the combined cohort including all 10
cancers, the DLS was significantly associated with disease specific survival
(hazard ratio of 1.58, 95% CI 1.28-1.70, p<0.0001) after adjusting for cancer
type, stage, age, and sex. In a per-cancer adjusted subanalysis, the DLS
remained a significant predictor of survival in 5 of 10 cancer types. Compared
to a baseline model including stage, age, and sex, the c-index of the model
demonstrated an absolute 3.7% improvement (95% CI 1.0-6.5) in the combined
cohort. Additionally, our models stratified patients within individual cancer
stages, particularly stage II (p=0.025) and stage III (p<0.001). By developing
and evaluating prognostic models across multiple cancer types, this work
represents one of the most comprehensive studies exploring the direct
prediction of clinical outcomes using deep learning and histopathology images.
Our analysis demonstrates the potential for this approach to provide prognostic
information in multiple cancer types, and even within specific pathologic
stages. However, given the relatively small number of clinical events, we
observed wide confidence intervals, suggesting that future work will benefit
from larger datasets.

通过开发深度学习系统，在多种癌症类型中利用组织病理图像预测疾病特异性生存，为癌症治疗和监测提供重要的预后信息。