We present a self-supervised learning approach to learn audio-visual
representations from video and audio. Our method uses contrastive learning for
cross-modal discrimination of video from audio and vice-versa. We show that
optimizing for cross-modal discrimination, rather than within-modal
discrimination, is important to learn good representations from video and
audio. With this simple but powerful insight, our method achieves highly
competitive performance when finetuned on action recognition tasks.
Furthermore, while recent work in contrastive learning defines positive and
negative samples as individual instances, we generalize this definition by
exploring cross-modal agreement. We group together multiple instances as
positives by measuring their similarity in both the video and audio feature
spaces. Cross-modal agreement creates better positive and negative sets, which
allows us to calibrate visual similarities by seeking within-modal
discrimination of positive instances, and achieve significant gains on
downstream tasks.

文中介绍了一种自监督学习方法，通过交叉模态辨别视频和音频来学习视听表征，其采用对比学习方法来进行。该方法通过优化交叉模态辨别而非内部模态辨别，可以学习到来自视频和音频的良好表征，并可于行动识别任务中获得高竞争性能。同时，文中提到该方法采用交叉模态协同度量相似性，构造更好的正负样本集合，从而实现了对可视相似度的校准。