Technological advances in medical data collection such as high-resolution histopathology and high-throughput genomic sequencing have contributed to the rising requirement for multi-modal biomedical modelling, specifically for image, tabular, and graph data. Most multi-modal deep learning approaches use modality-specific architectures that are trained separately and cannot capture the crucial cross-modal information that motivates the integration of different data sources. This paper presents the Hybrid Early-fusion Attention Learning Network (HEALNet): a flexible multi-modal fusion architecture, which a) preserves modality-specific structural information, b) captures the cross-modal interactions and structural information in a shared latent space, c) can effectively handle missing modalities during training and inference, and d) enables intuitive model inspection by learning on the raw data input instead of opaque embeddings. We conduct multi-modal survival analysis on Whole Slide Images and Multi-omic data on four cancer cohorts of The Cancer Genome Atlas (TCGA). HEALNet achieves state-of-the-art performance, substantially improving over both uni-modal and recent multi-modal baselines, whilst being robust in scenarios with missing modalities.

采用新技术对医学数据进行收集，如高分辨率组织病理学和高通量基因组测序，为多模态生物医学建模提供了支持。本文介绍了一种灵活的多模态融合架构——混合早期融合注意力学习网络（HEALNet），能够有效处理训练和推断中缺失的模态，提供了最新的多模态基准，并在存在缺失模态的场景中获得卓越性能。

HEALNet -- 混合多模态融合为异构生物医学数据