The objective of Audio-Visual Segmentation (AVS) is to locate sounding
objects within visual scenes by accurately predicting pixelwise segmentation
masks. In this paper, we present the following contributions: (i), we propose a
scalable and annotation-free pipeline for generating artificial data for the
AVS task. We leverage existing image segmentation and audio datasets to draw
links between category labels, image-mask pairs, and audio samples, which
allows us to easily compose (image, audio, mask) triplets for training AVS
models; (ii), we introduce a novel Audio-Aware Transformer (AuTR) architecture
that features an audio-aware query-based transformer decoder. This architecture
enables the model to search for sounding objects with the guidance of audio
signals, resulting in more accurate segmentation; (iii), we present extensive
experiments conducted on both synthetic and real datasets, which demonstrate
the effectiveness of training AVS models with synthetic data generated by our
proposed pipeline. Additionally, our proposed AuTR architecture exhibits
superior performance and strong generalization ability on public benchmarks.
The project page is this https URL

本文提出了一种用于定位视觉场景中声音对象的 Audio-Visual Segmentation (AVS) 的方法，其中使用了缩放和无注释的管道来生成 AVS 模型的合成数据，还提出了一种 Audio-Aware Transformer (AuTR) 结构，具有音频感知的查询式变压器解码器，以使模型更准确地进行分割。作者在合成和实际数据集上进行了广泛的实验，并取得了很好的效果。