Deep learning approaches, together with neuroimaging techniques, play an
important role in psychiatric disorders classification. Previous studies on
psychiatric disorders diagnosis mainly focus on using functional connectivity
matrices of resting-state functional magnetic resonance imaging (rs-fMRI) as
input, which still needs to fully utilize the rich temporal information of the
time series of rs-fMRI data. In this work, we proposed a
multi-dimension-embedding-aware modality fusion transformer (MFFormer) for
schizophrenia and bipolar disorder classification using rs-fMRI and T1 weighted
structural MRI (T1w sMRI). Concretely, to fully utilize the temporal
information of rs-fMRI and spatial information of sMRI, we constructed a deep
learning architecture that takes as input 2D time series of rs-fMRI and 3D
volumes T1w. Furthermore, to promote intra-modality attention and information
fusion across different modalities, a fusion transformer module (FTM) is
designed through extensive self-attention of hybrid feature maps of
multi-modality. In addition, a dimension-up and dimension-down strategy is
suggested to properly align feature maps of multi-dimensional from different
modalities. Experimental results on our private and public OpenfMRI datasets
show that our proposed MFFormer performs better than that using a single
modality or multi-modality MRI on schizophrenia and bipolar disorder diagnosis.

我们提出了一种基于多维嵌入感知的模态融合变压器（MFFormer）来使用静息状态功能磁共振成像（rs-fMRI）和 T1 加权结构磁共振成像（T1w sMRI）对精神分裂症和双相障碍进行分类，实验结果表明我们的 MFFormer 在精神分裂症和双相障碍诊断上表现优于单一或多模态 MRI。

多维嵌入感知的模态融合变压器用于精神障碍分类

Multi-Dimension-Embedding-Aware Modality Fusion Transformer for  Psychiatric Disorder Clasification

Magnetic Resonance Imaging (MRI) typically recruits multiple sequences
(defined here as "modalities"). As each modality is designed to offer different
anatomical and functional clinical information, there are evident disparities
in the imaging content across modalities. Inter- and intra-modality affine and
non-rigid image registration is an essential medical image analysis process in
clinical imaging, as for example before imaging biomarkers need to be derived
and clinically evaluated across different MRI modalities, time phases and
slices. Although commonly needed in real clinical scenarios, affine and
non-rigid image registration is not extensively investigated using a single
unsupervised model architecture. In our work, we present an un-supervised deep
learning registration methodology which can accurately model affine and
non-rigid trans-formations, simultaneously. Moreover, inverse-consistency is a
fundamental inter-modality registration property that is not considered in deep
learning registration algorithms. To address inverse-consistency, our
methodology performs bi-directional cross-modality image synthesis to learn
modality-invariant latent rep-resentations, while involves two factorised
transformation networks and an inverse-consistency loss to learn
topology-preserving anatomical transformations. Overall, our model (named
"FIRE") shows improved performances against the reference standard baseline
method on multi-modality brain 2D and 3D MRI and intra-modality cardiac 4D MRI
data experiments.

本文提出了一种无监督深度学习配准方法，采用双向交叉模态图像合成提高正逆一致性，实现了跨不同 MRI 模态、时间和层面的解剖学变换，获得了优于基准方法的多模态脑部 2D 和 3D MRI 以及单模态心脏 4D MRI 数据实验结果。