Medical multi-modal pre-training has revealed promise in computer-aided
diagnosis by leveraging large-scale unlabeled datasets. However, existing
methods based on masked autoencoders mainly rely on data-level reconstruction
tasks, but lack high-level semantic information. Furthermore, two significant
heterogeneity challenges hinder the transfer of pre-trained knowledge to
downstream tasks, \textit{i.e.}, the distribution heterogeneity between
pre-training data and downstream data, and the modality heterogeneity within
downstream data. To address these challenges, we propose a Unified Medical
Multi-modal Diagnostic (UMD) framework with tailored pre-training and
downstream tuning strategies. Specifically, to enhance the representation
abilities of vision and language encoders, we propose the Multi-level
Reconstruction Pre-training (MR-Pretrain) strategy, including a feature-level
and data-level reconstruction, which guides models to capture the semantic
information from masked inputs of different modalities. Moreover, to tackle two
kinds of heterogeneities during the downstream tuning, we present the
heterogeneity-combat downstream tuning strategy, which consists of a
Task-oriented Distribution Calibration (TD-Calib) and a Gradient-guided
Modality Coordination (GM-Coord). In particular, TD-Calib fine-tunes the
pre-trained model regarding the distribution of downstream datasets, and
GM-Coord adjusts the gradient weights according to the dynamic optimization
status of different modalities. Extensive experiments on five public medical
datasets demonstrate the effectiveness of our UMD framework, which remarkably
outperforms existing approaches on three kinds of downstream tasks.

提出了一种统一的医学多模态诊断 (UMD) 框架，包括量身定制的预训练和下游调优策略，以解决医学多模态预训练面临的数据和模态异质性挑战，在五个公共医学数据集上的广泛实验表明我们的 UMD 框架在三种下游任务上显著优于现有方法。