Multi-modal learning is typically performed with network architectures
containing modality-specific layers and shared layers, utilizing co-registered
images of different modalities. We propose a novel learning scheme for unpaired
cross-modality image segmentation, with a highly compact architecture achieving
superior segmentation accuracy. In our method, we heavily reuse network
parameters, by sharing all convolutional kernels across CT and MRI, and only
employ modality-specific internal normalization layers which compute respective
statistics. To effectively train such a highly compact model, we introduce a
novel loss term inspired by knowledge distillation, by explicitly constraining
the KL-divergence of our derived prediction distributions between modalities.
We have extensively validated our approach on two multi-class segmentation
problems: i) cardiac structure segmentation, and ii) abdominal organ
segmentation. Different network settings, i.e., 2D dilated network and 3D
U-net, are utilized to investigate our method's general efficacy. Experimental
results on both tasks demonstrate that our novel multi-modal learning scheme
consistently outperforms single-modal training and previous multi-modal
approaches.

本研究提出了一种新颖的多模式学习方案，以实现无配对交叉模态图像分割，该方案采用高度压缩的体系结构，通过共享所有卷积核跨 CT 和 MRI 和仅使用特定于模态的内部归一化层来对网络参数进行大量重用，通过知识蒸馏受启发的新颖损失项来显式约束我们推导出的模态之间的预测分布的 KL 散度来有效地训练这样的高度紧凑模型。在心脏结构分割和腹部器官分割两个多类分割问题上进行了广泛验证，实验结果表明，我们的新型多模式学习方案在两个任务上一致优于单模态培训和以前的多模式方法。