We introduce MHVAE, a deep hierarchical variational auto-encoder (VAE) that
synthesizes missing images from various modalities. Extending multi-modal VAEs
with a hierarchical latent structure, we introduce a probabilistic formulation
for fusing multi-modal images in a common latent representation while having
the flexibility to handle incomplete image sets as input. Moreover, adversarial
learning is employed to generate sharper images. Extensive experiments are
performed on the challenging problem of joint intra-operative ultrasound (iUS)
and Magnetic Resonance (MR) synthesis. Our model outperformed multi-modal VAEs,
conditional GANs, and the current state-of-the-art unified method (ResViT) for
synthesizing missing images, demonstrating the advantage of using a
hierarchical latent representation and a principled probabilistic fusion
operation. Our code is publicly available
https://github.com/ReubenDo/MHVAE.

MHVAE 是一种深度分层变分自动编码器，用于合成不同模态的缺失图像，并可处理输入的不完整图像集。通过引入分层潜在结构的多模态 VAE 扩展，我们采用概率模型融合了多模态图像到共同的潜在表示，并采用对抗学习生成更清晰的图像。我们在关节术中超声和磁共振图像合成的具有挑战性的问题上进行了广泛实验。相对于多模态 VAE、条件 GAN 和当前最先进的统一方法（ResViT），我们的模型表现更好，显示了使用分层潜在表示和基于原则的概率融合操作的优势。我们的代码公开可用于 https://github.com/ReubenDo/MHVAE。