Multi-modal image registration is a challenging problem that is also an
important clinical task for many real applications and scenarios. As a first
step in analysis, deformable registration among different image modalities is
often required in order to provide complementary visual information. During
registration, semantic information is key to match homologous points and
pixels. Nevertheless, many conventional registration methods are incapable in
capturing high-level semantic anatomical dense correspondences. In this work,
we propose a novel multi-task learning system, JSSR, based on an end-to-end 3D
convolutional neural network that is composed of a generator, a registration
and a segmentation component. The system is optimized to satisfy the implicit
constraints between different tasks in an unsupervised manner. It first
synthesizes the source domain images into the target domain, then an
intra-modal registration is applied on the synthesized images and target
images. The segmentation module are then applied on the synthesized and target
images, providing additional cues based on semantic correspondences. The
supervision from another fully-annotated dataset is used to regularize the
segmentation. We extensively evaluate JSSR on a large-scale medical image
dataset containing 1,485 patient CT imaging studies of four different contrast
phases (i.e., 5,940 3D CT scans with pathological livers) on the registration,
segmentation and synthesis tasks. The performance is improved after joint
training on the registration and segmentation tasks by 0.9% and 1.9%
respectively compared to a highly competitive and accurate deep learning
baseline. The registration also consistently outperforms conventional
state-of-the-art multi-modal registration methods.

本文提出了一种基于端到端 3D 卷积神经网络的多任务学习系统 JSSR，用于医学图像的多模态图像配准、分割和合成，该方法能够有效捕获高级语义密集对应，优于市场上已有的相关方法。