Sound source localization in visual scenes aims to localize objects emitting
the sound in a given image. Recent works showing impressive localization
performance typically rely on the contrastive learning framework. However, the
random sampling of negatives, as commonly adopted in these methods, can result
in misalignment between audio and visual features and thus inducing ambiguity
in localization. In this paper, instead of following previous literature, we
propose Self-Supervised Predictive Learning (SSPL), a negative-free method for
sound localization via explicit positive mining. Specifically, we first devise
a three-stream network to elegantly associate sound source with two augmented
views of one corresponding video frame, leading to semantically coherent
similarities between audio and visual features. Second, we introduce a novel
predictive coding module for audio-visual feature alignment. Such a module
assists SSPL to focus on target objects in a progressive manner and effectively
lowers the positive-pair learning difficulty. Experiments show surprising
results that SSPL outperforms the state-of-the-art approach on two standard
sound localization benchmarks. In particular, SSPL achieves significant
improvements of 8.6% cIoU and 3.4% AUC on SoundNet-Flickr compared to the
previous best. Code is available at: this https URL

本文提出了自监督预测学习 (SSPL) 方法，通过显式正样本挖掘以实现声音定位，将声音来源与视频帧的两个增强视图相结合，并引入了预测编码模块以帮助 SSPL 逐步聚焦目标对象和有效降低正向对难度。实验结果表明，SSPL 在两个标准的声音定位基准测试中优于现有最佳方法，在 SoundNet-Flickr 上将 cIoU 和 AUC 分别提高了 8.6％和 3.4％。