Multimodal Magnetic Resonance (MR) Imaging plays a crucial role in disease
diagnosis due to its ability to provide complementary information by analyzing
a relationship between multimodal images on the same subject. Acquiring all MR
modalities, however, can be expensive, and, during a scanning session, certain
MR images may be missed depending on the study protocol. The typical solution
would be to synthesize the missing modalities from the acquired images such as
using generative adversarial networks (GANs). Yet, GANs constructed with
convolutional neural networks (CNNs) are likely to suffer from a lack of global
relationships and mechanisms to condition the desired modality. To address
this, in this work, we propose a transformer-based modality infuser designed to
synthesize multimodal brain MR images. In our method, we extract
modality-agnostic features from the encoder and then transform them into
modality-specific features using the modality infuser. Furthermore, the
modality infuser captures long-range relationships among all brain structures,
leading to the generation of more realistic images. We carried out experiments
on the BraTS 2018 dataset, translating between four MR modalities, and our
experimental results demonstrate the superiority of our proposed method in
terms of synthesis quality. In addition, we conducted experiments on a brain
tumor segmentation task and different conditioning methods.

本研究提出了一个基于 Transformer 的模态注入器，用于合成多模态脑 MR 图像，通过从编码器中提取模态不可知特征，然后将其转换为模态特定特征，进而捕捉所有脑结构之间的远距离关系，从而生成更真实的图像。在 BraTS 2018 数据集上进行实验的结果表明，所提出的方法在合成质量方面具有优势。此外，还对脑肿瘤分割任务和不同的条件方法进行了实验。

基于 Transformer 的多模态脑 MR 影像分解转换器

Disentangled Multimodal Brain MR Image Translation via Transformer-based  Modality Infuser

Multimodal magnetic resonance imaging (MRI) can reveal different patterns of
human tissue and is crucial for clinical diagnosis. However, limited by cost,
noise and manual labeling, obtaining diverse and reliable multimodal MR images
remains a challenge. For the same lesion, different MRI manifestations have
great differences in background information, coarse positioning and fine
structure. In order to obtain better generation and segmentation performance, a
coordination-spatial attention generation adversarial network (CASP-GAN) based
on the cycle-consistent generative adversarial network (CycleGAN) is proposed.
The performance of the generator is optimized by introducing the Coordinate
Attention (CA) module and the Spatial Attention (SA) module. The two modules
can make full use of the captured location information, accurately locating the
interested region, and enhancing the generator model network structure. The
ability to extract the structure information and the detailed information of
the original medical image can help generate the desired image with higher
quality. There exist some problems in the original CycleGAN that the training
time is long, the parameter amount is too large, and it is difficult to
converge. In response to this problem, we introduce the Coordinate Attention
(CA) module to replace the Res Block to reduce the number of parameters, and
cooperate with the spatial information extraction network above to strengthen
the information extraction ability. On the basis of CASP-GAN, an attentional
generative cross-modality segmentation (AGCMS) method is further proposed. This
method inputs the modalities generated by CASP-GAN and the real modalities into
the segmentation network for brain tumor segmentation. Experimental results
show that CASP-GAN outperforms CycleGAN and some state-of-the-art methods in
PSNR, SSMI and RMSE in most tasks.

提出了一种基于 CASP-GAN 的协调空间注意力生成对抗网络，以及一种关注性生成跨模态分割方法，该方法在脑肿瘤分割中取得了比 CycleGAN 和部分前沿方法更好的结果。