Deep learning approaches, together with neuroimaging techniques, play an
important role in psychiatric disorders classification. Previous studies on
psychiatric disorders diagnosis mainly focus on using functional connectivity
matrices of resting-state functional magnetic resonance imaging (rs-fMRI) as
input, which still needs to fully utilize the rich temporal information of the
time series of rs-fMRI data. In this work, we proposed a
multi-dimension-embedding-aware modality fusion transformer (MFFormer) for
schizophrenia and bipolar disorder classification using rs-fMRI and T1 weighted
structural MRI (T1w sMRI). Concretely, to fully utilize the temporal
information of rs-fMRI and spatial information of sMRI, we constructed a deep
learning architecture that takes as input 2D time series of rs-fMRI and 3D
volumes T1w. Furthermore, to promote intra-modality attention and information
fusion across different modalities, a fusion transformer module (FTM) is
designed through extensive self-attention of hybrid feature maps of
multi-modality. In addition, a dimension-up and dimension-down strategy is
suggested to properly align feature maps of multi-dimensional from different
modalities. Experimental results on our private and public OpenfMRI datasets
show that our proposed MFFormer performs better than that using a single
modality or multi-modality MRI on schizophrenia and bipolar disorder diagnosis.

我们提出了一种基于多维嵌入感知的模态融合变压器（MFFormer）来使用静息状态功能磁共振成像（rs-fMRI）和 T1 加权结构磁共振成像（T1w sMRI）对精神分裂症和双相障碍进行分类，实验结果表明我们的 MFFormer 在精神分裂症和双相障碍诊断上表现优于单一或多模态 MRI。