Audiovisual representation learning typically relies on the correspondence
between sight and sound. However, there are often multiple audio tracks that
can correspond with a visual scene. Consider, for example, different
conversations on the same crowded street. The effect of such counterfactual
pairs on audiovisual representation learning has not been previously explored.
To investigate this, we use dubbed versions of movies to augment cross-modal
contrastive learning. Our approach learns to represent alternate audio tracks,
differing only in speech content, similarly to the same video. Our results show
that dub-augmented training improves performance on a range of auditory and
audiovisual tasks, without significantly affecting linguistic task performance
overall. We additionally compare this approach to a strong baseline where we
remove speech before pretraining, and find that dub-augmented training is more
effective, including for paralinguistic and audiovisual tasks where speech
removal leads to worse performance. These findings highlight the importance of
considering speech variation when learning scene-level audiovisual
correspondences and suggest that dubbed audio can be a useful augmentation
technique for training audiovisual models toward more robust performance.

研究了在音频与视觉之间进行对应时，出现多个音频轨道时的学习效果，探讨了使用配音版本来增加跨模态对比学习的方法，提出了考虑语音变化时学习场景级别的音频视觉对应关系的重要性，并表明配音可以作为训练音频视觉模型的一种有用增强技术。