Multimodal learning, integrating histology images and genomics, promises to
enhance precision oncology with comprehensive views at microscopic and
molecular levels. However, existing methods may not sufficiently model the
shared or complementary information for more effective integration. In this
study, we introduce a Unified Modeling Enhanced Multimodal Learning (UMEML)
framework that employs a hierarchical attention structure to effectively
leverage shared and complementary features of both modalities of histology and
genomics. Specifically, to mitigate unimodal bias from modality imbalance, we
utilize a query-based cross-attention mechanism for prototype clustering in the
pathology encoder. Our prototype assignment and modularity strategy are
designed to align shared features and minimizes modality gaps. An additional
registration mechanism with learnable tokens is introduced to enhance
cross-modal feature integration and robustness in multimodal unified modeling.
Our experiments demonstrate that our method surpasses previous state-of-the-art
approaches in glioma diagnosis and prognosis tasks, underscoring its
superiority in precision neuro-Oncology.

通过使用一种层次化注意结构，我们介绍了一种统一建模增强多模态学习（UMEML）框架，该框架整合了组织学图像和基因组学，以在显微镜级别和分子级别上实现对肿瘤的精确治疗，并通过利用两种模态的共享和互补特征有效地进行建模。