We propose to explore a new problem called audio-visual segmentation (AVS),
in which the goal is to output a pixel-level map of the object(s) that produce
sound at the time of the image frame. To facilitate this research, we construct
the first audio-visual segmentation benchmark (AVSBench), providing pixel-wise
annotations for the sounding objects in audible videos. Two settings are
studied with this benchmark: 1) semi-supervised audio-visual segmentation with
a single sound source and 2) fully-supervised audio-visual segmentation with
multiple sound sources. To deal with the AVS problem, we propose a novel method
that uses a temporal pixel-wise audio-visual interaction module to inject audio
semantics as guidance for the visual segmentation process. We also design a
regularization loss to encourage the audio-visual mapping during training.
Quantitative and qualitative experiments on the AVSBench compare our approach
to several existing methods from related tasks, demonstrating that the proposed
method is promising for building a bridge between the audio and pixel-wise
visual semantics. Code is available at this https URL

本文提出音频与视觉分割（AVS）问题，并通过构建 AVSBench 基准集进行了研究。通过引入一种新颖的方法 —— 时间上的像素级音视频交互模块，可以指导视觉分割过程，同时设计了正则化损失函数以鼓励音视频映射的训练，比较了几种现有方法，发现该方法有望在音频和像素级视觉语义之间搭建桥梁。