Medical multi-modal pre-training has revealed promise in computer-aided
diagnosis by leveraging large-scale unlabeled datasets. However, existing
methods based on masked autoencoders mainly rely on data-level reconstruction
tasks, but lack high-level semantic information. Furthermore, two significant
heterogeneity challenges hinder the transfer of pre-trained knowledge to
downstream tasks, \textit{i.e.}, the distribution heterogeneity between
pre-training data and downstream data, and the modality heterogeneity within
downstream data. To address these challenges, we propose a Unified Medical
Multi-modal Diagnostic (UMD) framework with tailored pre-training and
downstream tuning strategies. Specifically, to enhance the representation
abilities of vision and language encoders, we propose the Multi-level
Reconstruction Pre-training (MR-Pretrain) strategy, including a feature-level
and data-level reconstruction, which guides models to capture the semantic
information from masked inputs of different modalities. Moreover, to tackle two
kinds of heterogeneities during the downstream tuning, we present the
heterogeneity-combat downstream tuning strategy, which consists of a
Task-oriented Distribution Calibration (TD-Calib) and a Gradient-guided
Modality Coordination (GM-Coord). In particular, TD-Calib fine-tunes the
pre-trained model regarding the distribution of downstream datasets, and
GM-Coord adjusts the gradient weights according to the dynamic optimization
status of different modalities. Extensive experiments on five public medical
datasets demonstrate the effectiveness of our UMD framework, which remarkably
outperforms existing approaches on three kinds of downstream tasks.

提出了一种统一的医学多模态诊断 (UMD) 框架，包括量身定制的预训练和下游调优策略，以解决医学多模态预训练面临的数据和模态异质性挑战，在五个公共医学数据集上的广泛实验表明我们的 UMD 框架在三种下游任务上显著优于现有方法。

具有重构预训练和异质性抗干扰调节的统一多模式诊断框架

Unified Multi-modal Diagnostic Framework with Reconstruction  Pre-training and Heterogeneity-combat Tuning

Recently, Federated Graph Learning (FGL) has attracted significant attention
as a distributed framework based on graph neural networks, primarily due to its
capability to break data silos. Existing FGL studies employ community split on
the homophilous global graph by default to simulate federated semi-supervised
node classification settings. Such a strategy assumes the consistency of
topology between the multi-client subgraphs and the global graph, where
connected nodes are highly likely to possess similar feature distributions and
the same label. However, in real-world implementations, the varying
perspectives of local data engineering result in various subgraph topologies,
posing unique heterogeneity challenges in FGL. Unlike the well-known label
Non-independent identical distribution (Non-iid) problems in federated
learning, FGL heterogeneity essentially reveals the topological divergence
among multiple clients, namely homophily or heterophily. To simulate and handle
this unique challenge, we introduce the concept of structure Non-iid split and
then present a new paradigm called \underline{Ada}ptive \underline{F}ederated
\underline{G}raph \underline{L}earning (AdaFGL), a decoupled two-step
personalized approach. To begin with, AdaFGL employs standard multi-client
federated collaborative training to acquire the federated knowledge extractor
by aggregating uploaded models in the final round at the server. Then, each
client conducts personalized training based on the local subgraph and the
federated knowledge extractor. Extensive experiments on the 12 graph benchmark
datasets validate the superior performance of AdaFGL over state-of-the-art
baselines. Specifically, in terms of test accuracy, our proposed AdaFGL
outperforms baselines by significant margins of 3.24\% and 5.57\% on community
split and structure Non-iid split, respectively.

AdaFGL 是一个分布式的基于图神经网络的框架，通过引入 structure Non-iid split 的概念，提供了一种解决 Federated Graph Learning 中独特异构性挑战的新范式，实验证实其在 12 个图网络基准数据集上具有优异的性能。