Discriminatively localizing sounding objects in cocktail-party, i.e., mixed
sound scenes, is commonplace for humans, but still challenging for machines. In
this paper, we propose a two-stage learning framework to perform
self-supervised class-aware sounding object localization. First, we propose to
learn robust object representations by aggregating the candidate sound
localization results in the single source scenes. Then, class-aware object
localization maps are generated in the cocktail-party scenarios by referring
the pre-learned object knowledge, and the sounding objects are accordingly
selected by matching audio and visual object category distributions, where the
audiovisual consistency is viewed as the self-supervised signal. Experimental
results in both realistic and synthesized cocktail-party videos demonstrate
that our model is superior in filtering out silent objects and pointing out the
location of sounding objects of different classes. Code is available at
this https URL

本研究提出了一种两阶段学习框架，利用单一场景下的候选声音定位结果来学习鲁棒的对象表示，并通过引用预学习的对象知识生成了类感知对象本地化映射，在各种混音声音场景下选择声音和视觉对象类别分布的匹配，其中视听一致性被视为自我监督信号。实验结果表明，我们的模型在过滤无声对象和指出不同类别声音对象位置方面优于其他方法。