State-of-the-art approaches for visually-guided audio source separation
typically assume sources that have characteristic sounds, such as musical
instruments. These approaches often ignore the visual context of these sound
sources or avoid modeling object interactions that may be useful to better
characterize the sources, especially when the same object class may produce
varied sounds from distinct interactions. To address this challenging problem,
we propose Audio Visual Scene Graph Segmenter (AVSGS), a novel deep learning
model that embeds the visual structure of the scene as a graph and segments
this graph into subgraphs, each subgraph being associated with a unique sound
obtained by co-segmenting the audio spectrogram. At its core, AVSGS uses a
recursive neural network that emits mutually-orthogonal sub-graph embeddings of
the visual graph using multi-head attention. These embeddings are used for
conditioning an audio encoder-decoder towards source separation. Our pipeline
is trained end-to-end via a self-supervised task consisting of separating audio
sources using the visual graph from artificially mixed sounds. In this paper,
we also introduce an "in the wild'' video dataset for sound source separation
that contains multiple non-musical sources, which we call Audio Separation in
the Wild (ASIW). This dataset is adapted from the AudioCaps dataset, and
provides a challenging, natural, and daily-life setting for source separation.
Thorough experiments on the proposed ASIW and the standard MUSIC datasets
demonstrate state-of-the-art sound separation performance of our method against
recent prior approaches.

本文提出了一种基于 Audio Visual Scene Graph Segmenter (AVSGS) 的深度学习模型，通过嵌入场景的视觉结构，并将其分割为子图，实现音频源分离；同时，介绍了一个全新的数据集 Audio Separation in the Wild (ASIW)，证明了该方法在音源分离方面的卓越表现。