Simultaneous functional PET/MR (sf-PET/MR) presents a cutting-edge multimodal
neuroimaging technique. It provides an unprecedented opportunity for
concurrently monitoring and integrating multifaceted brain networks built by
spatiotemporally covaried metabolic activity, neural activity, and cerebral
blood flow (perfusion). Albeit high scientific/clinical values, short in
hardware accessibility of PET/MR hinders its applications, let alone modern
AI-based PET/MR fusion models. Our objective is to develop a clinically
feasible AI-based disease diagnosis model trained on comprehensive sf-PET/MR
data with the power of, during inferencing, allowing single modality input
(e.g., PET only) as well as enforcing multimodal-based accuracy. To this end,
we propose MX-ARM, a multimodal MiXture-of-experts Alignment and Reconstruction
Model. It is modality detachable and exchangeable, allocating different
multi-layer perceptrons dynamically ("mixture of experts") through learnable
weights to learn respective representations from different modalities. Such
design will not sacrifice model performance in uni-modal situation. To fully
exploit the inherent complex and nonlinear relation among modalities while
producing fine-grained representations for uni-modal inference, we subsequently
add a modal alignment module to line up a dominant modality (e.g., PET) with
representations of auxiliary modalities (MR). We further adopt multimodal
reconstruction to promote the quality of learned features. Experiments on
precious multimodal sf-PET/MR data for Mild Cognitive Impairment diagnosis
showcase the efficacy of our model toward clinically feasible precision
medicine.

提出了 MX-ARM，一种多模态混合专家对齐与重建模型，通过学习可调权重的不同多层感知器（“专家混合”），动态地从不同模态学习相应的特征表示。实验证明该模型在诊断轻度认知障碍的多模态 sf-PET/MR 数据上具有临床可行的精准医学效果。