Recent progress in deep learning has enabled many advances in sound
separation and visual scene understanding. However, extracting sound sources
which are apparent in natural videos remains an open problem. In this work, we
present AudioScope, a novel audio-visual sound separation framework that can be
trained without supervision to isolate on-screen sound sources from real
in-the-wild videos. Prior audio-visual separation work assumed artificial
limitations on the domain of sound classes (e.g., to speech or music),
constrained the number of sources, and required strong sound separation or
visual segmentation labels. AudioScope overcomes these limitations, operating
on an open domain of sounds, with variable numbers of sources, and without
labels or prior visual segmentation. The training procedure for AudioScope uses
mixture invariant training (MixIT) to separate synthetic mixtures of mixtures
(MoMs) into individual sources, where noisy labels for mixtures are provided by
an unsupervised audio-visual coincidence model. Using the noisy labels, along
with attention between video and audio features, AudioScope learns to identify
audio-visual similarity and to suppress off-screen sounds. We demonstrate the
effectiveness of our approach using a dataset of video clips extracted from
open-domain YFCC100m video data. This dataset contains a wide diversity of
sound classes recorded in unconstrained conditions, making the application of
previous methods unsuitable. For evaluation and semi-supervised experiments, we
collected human labels for presence of on-screen and off-screen sounds on a
small subset of clips.

本研究提出了一种名为 AudioScope 的音频 - 视觉分离框架，在无监督培训下从自然视频中分离屏幕上的声源。使用混合不变训练（MixIT）对混合物进行训练并在电影与音频特征之间调用注意力来识别音频 - 视觉相似性并抑制屏幕外声音。