Auditory and visual signals usually present together and correlate with each
other, not only in natural environments but also in clinical settings. However,
the audio-visual modelling in the latter case can be more challenging, due to
the different sources of audio/video signals and the noise (both signal-level
and semantic-level) in auditory signals -- usually speech. In this paper, we
consider audio-visual modelling in a clinical setting, providing a solution to
learn medical representations that benefit various clinical tasks, without
human expert annotation. A simple yet effective multi-modal self-supervised
learning framework is proposed for this purpose. The proposed approach is able
to localise anatomical regions of interest during ultrasound imaging, with only
speech audio as a reference. Experimental evaluations on a large-scale clinical
multi-modal ultrasound video dataset show that the proposed self-supervised
method learns good transferable anatomical representations that boost the
performance of automated downstream clinical tasks, even outperforming
fully-supervised solutions.

本文介绍了一种在临床环境中的音频 - 视觉建模方法，提供了一种无需人工专家标注的学习医学表征的解决方案，通过只有语音音频作为参考，可以定位超声成像中感兴趣的解剖区域。在大规模临床多模式超声视频数据集上的实验评估表明，该自我监督方法学习到的良好可迁移的解剖表征可以提高自动化的下游临床任务的性能，甚至超过完全监督的解决方案。