Cancer diagnosis, prognosis, and therapeutic response predictions are based
on morphological information from histology slides and molecular profiles from
genomic data. However, most deep learning-based objective outcome prediction
and grading paradigms are based on histology or genomics alone and do not make
use of the complementary information in an intuitive manner. In this work, we
propose Pathomic Fusion, an interpretable strategy for end-to-end multimodal
fusion of histology image and genomic (mutations, CNV, RNA-Seq) features for
survival outcome prediction. Our approach models pairwise feature interactions
across modalities by taking the Kronecker product of unimodal feature
representations and controls the expressiveness of each representation via a
gating-based attention mechanism. Following supervised learning, we are able to
interpret and saliently localize features across each modality, and understand
how feature importance shifts when conditioning on multimodal input. We
validate our approach using glioma and clear cell renal cell carcinoma datasets
from the Cancer Genome Atlas (TCGA), which contains paired whole-slide image,
genotype, and transcriptome data with ground truth survival and histologic
grade labels. In a 15-fold cross-validation, our results demonstrate that the
proposed multimodal fusion paradigm improves prognostic determinations from
ground truth grading and molecular subtyping, as well as unimodal deep networks
trained on histology and genomic data alone. The proposed method establishes
insight and theory on how to train deep networks on multimodal biomedical data
in an intuitive manner, which will be useful for other problems in medicine
that seek to combine heterogeneous data streams for understanding diseases and
predicting response and resistance to treatment.

本文提出了一种解释性的、端到端的多模态融合策略 —— Pathomic Fusion，它基于组织学图像和基因组学（突变、CNV、RNA-Seq）特征，模型化跨模式的特征交互，并通过基于门控的注意机制控制每个表示的表现力，以提高生存结果预测的准确性，从而有助于其他医学问题的解决。

Pathomic Fusion：融合组织学和基因组特征的癌症诊断和预后一体化框架

Pathomic Fusion: An Integrated Framework for Fusing Histopathology and  Genomic Features for Cancer Diagnosis and Prognosis

Nuclear segmentation and classification within Haematoxylin & Eosin stained
histology images is a fundamental prerequisite in the digital pathology
work-flow. The development of automated methods for nuclear segmentation and
classification enables the quantitative analysis of tens of thousands of nuclei
within a whole-slide pathology image, opening up possibilities of further
analysis of large-scale nuclear morphometry. However, automated nuclear
segmentation and classification is faced with a major challenge in that there
are several different types of nuclei, some of them exhibiting large
intra-class variability such as the tumour cells. Additionally, some of the
nuclei are often clustered together. To address these challenges, we present a
novel convolutional neural network for simultaneous nuclear segmentation and
classification that leverages the instance-rich information encoded within the
vertical and horizontal distances of nuclear pixels to their centres of mass.
These distances are then utilised to separate clustered nuclei, resulting in an
accurate segmentation, particularly in areas with overlapping instances. Then
for each segmented instance, the network predicts the type of nucleus via a
devoted up-sampling branch. We demonstrate state-of-the-art performance
compared to other methods on multiple independent multi-tissue histology image
datasets. As part of this work, we introduce a new dataset of Haematoxylin &
Eosin stained colorectal adenocarcinoma image tiles, containing 24,319
exhaustively annotated nuclei with associated class labels.

通过使用卷积神经网络将核数据分离和分类，进而对数字病理学的工作流程进行优化。