Multimodal Magnetic Resonance (MR) Imaging plays a crucial role in disease diagnosis due to its ability to provide complementary information by analyzing a relationship between multimodal images on the same subject. Acquiring all MR modalities, however, can be expensive, and, during a scanning session, certain MR images may be missed depending on the study protocol. The typical solution would be to synthesize the missing modalities from the acquired images such as using generative adversarial networks (GANs). Yet, GANs constructed with convolutional neural networks (CNNs) are likely to suffer from a lack of global relationships and mechanisms to condition the desired modality. To address this, in this work, we propose a transformer-based modality infuser designed to synthesize multimodal brain MR images. In our method, we extract modality-agnostic features from the encoder and then transform them into modality-specific features using the modality infuser. Furthermore, the modality infuser captures long-range relationships among all brain structures, leading to the generation of more realistic images. We carried out experiments on the BraTS 2018 dataset, translating between four MR modalities, and our experimental results demonstrate the superiority of our proposed method in terms of synthesis quality. In addition, we conducted experiments on a brain tumor segmentation task and different conditioning methods.

本研究提出了一个基于Transformer的模态注入器，用于合成多模态脑MR图像，通过从编码器中提取模态不可知特征，然后将其转换为模态特定特征，进而捕捉所有脑结构之间的远距离关系，从而生成更真实的图像。在BraTS 2018数据集上进行实验的结果表明，所提出的方法在合成质量方面具有优势。此外，还对脑肿瘤分割任务和不同的条件方法进行了实验。

基于Transformer的多模态脑MR影像分解转换器