Auditory and visual signals usually present together and correlate with each
other, not only in natural environments but also in clinical settings. However,
the audio-visual modelling in the latter case can be more challenging, due to
the different sources of audio/video signals and the noise (both signal-level
and semantic-level) in auditory signals -- usually speech. In this paper, we
consider audio-visual modelling in a clinical setting, providing a solution to
learn medical representations that benefit various clinical tasks, without
human expert annotation. A simple yet effective multi-modal self-supervised
learning framework is proposed for this purpose. The proposed approach is able
to localise anatomical regions of interest during ultrasound imaging, with only
speech audio as a reference. Experimental evaluations on a large-scale clinical
multi-modal ultrasound video dataset show that the proposed self-supervised
method learns good transferable anatomical representations that boost the
performance of automated downstream clinical tasks, even outperforming
fully-supervised solutions.

本文介绍了一种在临床环境中的音频 - 视觉建模方法，提供了一种无需人工专家标注的学习医学表征的解决方案，通过只有语音音频作为参考，可以定位超声成像中感兴趣的解剖区域。在大规模临床多模式超声视频数据集上的实验评估表明，该自我监督方法学习到的良好可迁移的解剖表征可以提高自动化的下游临床任务的性能，甚至超过完全监督的解决方案。

展示而非描述：临床场景中的音视建模

Show from Tell: Audio-Visual Modelling in Clinical Settings

Brain tissue segmentation is essential for neuroscience and clinical studies.
However, segmentation on longitudinal data is challenging due to dynamic brain
changes across the lifespan. Previous researches mainly focus on
self-supervision with regularizations and will lose longitudinal generalization
when fine-tuning on a specific age group. In this paper, we propose a dual
meta-learning paradigm to learn longitudinally consistent representations and
persist when fine-tuning. Specifically, we learn a plug-and-play feature
extractor to extract longitudinal-consistent anatomical representations by
meta-feature learning and a well-initialized task head for fine-tuning by
meta-initialization learning. Besides, two class-aware regularizations are
proposed to encourage longitudinal consistency. Experimental results on the
iSeg2019 and ADNI datasets demonstrate the effectiveness of our method. Our
code is available at this https URL

本文提出了一种双元元学习范式，通过元特征学习提取纵向一致的解剖学特征表示，并通过元初始化学习提供良好初始化的任务头进行微调。此外，提出了两种类感知的正则化方法，以鼓励纵向一致性。在 iSeg2019 和 ADNI 数据集上的实验结果证明了我们方法的有效性。