Cross-modality medical image synthesis is a critical topic and has the
potential to facilitate numerous applications in the medical imaging field.
Despite recent successes in deep-learning-based generative models, most current
medical image synthesis methods rely on generative adversarial networks and
suffer from notorious mode collapse and unstable training. Moreover, the 2D
backbone-driven approaches would easily result in volumetric inconsistency,
while 3D backbones are challenging and impractical due to the tremendous memory
cost and training difficulty. In this paper, we introduce a new paradigm for
volumetric medical data synthesis by leveraging 2D backbones and present a
diffusion-based framework, Make-A-Volume, for cross-modality 3D medical image
synthesis. To learn the cross-modality slice-wise mapping, we employ a latent
diffusion model and learn a low-dimensional latent space, resulting in high
computational efficiency. To enable the 3D image synthesis and mitigate
volumetric inconsistency, we further insert a series of volumetric layers in
the 2D slice-mapping model and fine-tune them with paired 3D data. This
paradigm extends the 2D image diffusion model to a volumetric version with a
slightly increasing number of parameters and computation, offering a principled
solution for generic cross-modality 3D medical image synthesis. We showcase the
effectiveness of our Make-A-Volume framework on an in-house SWI-MRA brain MRI
dataset and a public T1-T2 brain MRI dataset. Experimental results demonstrate
that our framework achieves superior synthesis results with volumetric
consistency.

基于扩散的框架 Make-A-Volume 用于体积式医学数据的交叉模态三维图像合成，通过利用 2D 骨干和插入一系列体积层，成功地解决了体积不一致和计算效率问题，实现了具有一致性的优质合成结果。

利用潜在扩散模型进行跨模态 3D 脑 MRI 合成

Make-A-Volume: Leveraging Latent Diffusion Models for Cross-Modality 3D  Brain MRI Synthesis

The problem of how to assess cross-modality medical image synthesis has been
largely unexplored. The most used measures like PSNR and SSIM focus on
analyzing the structural features but neglect the crucial lesion location and
fundamental k-space speciality of medical images. To overcome this problem, we
propose a new metric K-CROSS to spur progress on this challenging problem.
Specifically, K-CROSS uses a pre-trained multi-modality segmentation network to
predict the lesion location, together with a tumor encoder for representing
features, such as texture details and brightness intensities. To further
reflect the frequency-specific information from the magnetic resonance imaging
principles, both k-space features and vision features are obtained and employed
in our comprehensive encoders with a frequency reconstruction penalty. The
structure-shared encoders are designed and constrained with a similarity loss
to capture the intrinsic common structural information for both modalities. As
a consequence, the features learned from lesion regions, k-space, and
anatomical structures are all captured, which serve as our quality evaluators.
We evaluate the performance by constructing a large-scale cross-modality
neuroimaging perceptual similarity (NIRPS) dataset with 6,000 radiologist
judgments. Extensive experiments demonstrate that the proposed method
outperforms other metrics, especially in comparison with the radiologists on
NIRPS.

通过引入预训练的多模态分割网络用于预测病变位置，以及肿瘤编码器用于表示特征，如纹理细节和亮度强度，结合使用 k 空间特征和视觉特征的全面编码器和频率重建惩罚，采用结构共享编码器和相似性损失，本研究提出了一种新的指标 K-CROSS 用于评估跨模态医学图像合成，通过构建一个包含 6,000 名放射科医生判断的大规模跨模态神经影像感知相似度（NIRPS）数据集，广泛的实验证明了该方法相较于其他指标的性能优越，特别是在 NIRPS 上与放射科医生的比较中。