We propose a new problem called audio-visual segmentation (AVS), in which the
goal is to output a pixel-level map of the object(s) that produce sound at the
time of the image frame. To facilitate this research, we construct the first
audio-visual segmentation benchmark, i.e., AVSBench, providing pixel-wise
annotations for sounding objects in audible videos. It contains three subsets:
AVSBench-object (Single-source subset, Multi-sources subset) and
AVSBench-semantic (Semantic-labels subset). Accordingly, three settings are
studied: 1) semi-supervised audio-visual segmentation with a single sound
source; 2) fully-supervised audio-visual segmentation with multiple sound
sources, and 3) fully-supervised audio-visual semantic segmentation. The first
two settings need to generate binary masks of sounding objects indicating
pixels corresponding to the audio, while the third setting further requires
generating semantic maps indicating the object category. To deal with these
problems, we propose a new baseline method that uses a temporal pixel-wise
audio-visual interaction module to inject audio semantics as guidance for the
visual segmentation process. We also design a regularization loss to encourage
audio-visual mapping during training. Quantitative and qualitative experiments
on AVSBench compare our approach to several existing methods for related tasks,
demonstrating that the proposed method is promising for building a bridge
between the audio and pixel-wise visual semantics. Code is available at
this https URL Online benchmark is available at
this http URL

本论文提出了一种新的问题 —— 音视频分割（AVS），旨在为可听的视频帧中产生声音的物体输出像素级地图，并构建了第一个 audio-visual segmentation（AVS）基准，即 AVSBench，为声音对象提供像素级注释。通过使用一种基于时间的像素级音视频交互模块注入音频语义指导视觉分割过程并设计一种正则化损失来鼓励训练期间的音视频映射，实验表明我们的方法能够有效地解决这个问题。