Self-supervised learning approaches leverage unlabeled samples to acquire
generic knowledge about different concepts, hence allowing for
annotation-efficient downstream task learning. In this paper, we propose a
novel self-supervised method that leverages multiple imaging modalities. We
introduce the multimodal puzzle task, which facilitates rich representation
learning from multiple image modalities. The learned representations allow for
subsequent fine-tuning on different downstream tasks. To achieve that, we learn
a modality-agnostic feature embedding by confusing image modalities at the
data-level. Together with the Sinkhorn operator, with which we formulate the
puzzle solving optimization as permutation matrix inference instead of
classification, they allow for efficient solving of multimodal puzzles with
varying levels of complexity. In addition, we also propose to utilize
cross-modal generation techniques for multimodal data augmentation used for
training self-supervised tasks. In other words, we exploit synthetic images for
self-supervised pretraining, instead of downstream tasks directly, in order to
circumvent quality issues associated with synthetic images, while improving
data-efficiency and representations quality. Our experimental results, which
assess the gains in downstream performance and data-efficiency, show that
solving our multimodal puzzles yields better semantic representations, compared
to treating each modality independently. Our results also highlight the
benefits of exploiting synthetic images for self-supervised pretraining. We
showcase our approach on four downstream tasks: Brain tumor segmentation and
survival days prediction using four MRI modalities, Prostate segmentation using
two MRI modalities, and Liver segmentation using unregistered CT and MRI
modalities. We outperform many previous solutions, and achieve results
competitive to state-of-the-art.

该论文提出了一种利用多个成像模式的新型自监督方法，设计了一个多模式拼图任务以促进多种图像模式的特征表征，提出了一种利用跨模态生成技术进行多模态数据增强的方法，实验表明该方法能够提高数据效率和语义表征，广泛应用于四种不同的下游任务，并取得了领先的成果。