In medical imaging, manual annotations can be expensive to acquire and
sometimes infeasible to access, making conventional deep learning-based models
difficult to scale. As a result, it would be beneficial if useful
representations could be derived from raw data without the need for manual
annotations. In this paper, we propose to address the problem of
self-supervised representation learning with multi-modal ultrasound
video-speech raw data. For this case, we assume that there is a high
correlation between the ultrasound video and the corresponding narrative speech
audio of the sonographer. In order to learn meaningful representations, the
model needs to identify such correlation and at the same time understand the
underlying anatomical features. We designed a framework to model the
correspondence between video and audio without any kind of human annotations.
Within this framework, we introduce cross-modal contrastive learning and an
affinity-aware self-paced learning scheme to enhance correlation modelling.
Experimental evaluations on multi-modal fetal ultrasound video and audio show
that the proposed approach is able to learn strong representations and
transfers well to downstream tasks of standard plane detection and eye-gaze
prediction.

本文提出了一种用于医学图像处理的自监督学习方法，该方法可以从多通道超声视频和相应的言语音频数据中学习有意义的图像特征，评价结果表明该方法可以用于标准平面检测和视线预测等向下流任务。