Audio-visual segmentation is a challenging task that aims to predict
pixel-level masks for sound sources in a video. Previous work applied a
comprehensive manually designed architecture with countless pixel-wise accurate
masks as supervision. However, these pixel-level masks are expensive and not
available in all cases. In this work, we aim to simplify the supervision as the
instance-level annotation, i.e., weakly-supervised audio-visual segmentation.
We present a novel Weakly-Supervised Audio-Visual Segmentation framework,
namely WS-AVS, that can learn multi-scale audio-visual alignment with
multi-scale multiple-instance contrastive learning for audio-visual
segmentation. Extensive experiments on AVSBench demonstrate the effectiveness
of our WS-AVS in the weakly-supervised audio-visual segmentation of
single-source and multi-source scenarios.

本文介绍了一种名为 WS-AVS 的弱监督音视频分割框架，通过多尺度多实例对比学习实现了多尺度音视频对齐和音视频分割，在单一源和多源情景下有效地进行了弱监督音视频分割。

弱监督的音频 - 视觉分割

Weakly-Supervised Audio-Visual Segmentation

We propose a new problem called audio-visual segmentation (AVS), in which the
goal is to output a pixel-level map of the object(s) that produce sound at the
time of the image frame. To facilitate this research, we construct the first
audio-visual segmentation benchmark, i.e., AVSBench, providing pixel-wise
annotations for sounding objects in audible videos. It contains three subsets:
AVSBench-object (Single-source subset, Multi-sources subset) and
AVSBench-semantic (Semantic-labels subset). Accordingly, three settings are
studied: 1) semi-supervised audio-visual segmentation with a single sound
source; 2) fully-supervised audio-visual segmentation with multiple sound
sources, and 3) fully-supervised audio-visual semantic segmentation. The first
two settings need to generate binary masks of sounding objects indicating
pixels corresponding to the audio, while the third setting further requires
generating semantic maps indicating the object category. To deal with these
problems, we propose a new baseline method that uses a temporal pixel-wise
audio-visual interaction module to inject audio semantics as guidance for the
visual segmentation process. We also design a regularization loss to encourage
audio-visual mapping during training. Quantitative and qualitative experiments
on AVSBench compare our approach to several existing methods for related tasks,
demonstrating that the proposed method is promising for building a bridge
between the audio and pixel-wise visual semantics. Code is available at
this https URL Online benchmark is available at
this http URL

本论文提出了一种新的问题 —— 音视频分割（AVS），旨在为可听的视频帧中产生声音的物体输出像素级地图，并构建了第一个 audio-visual segmentation（AVS）基准，即 AVSBench，为声音对象提供像素级注释。通过使用一种基于时间的像素级音视频交互模块注入音频语义指导视觉分割过程并设计一种正则化损失来鼓励训练期间的音视频映射，实验表明我们的方法能够有效地解决这个问题。

带语义的音视频分割

Audio-Visual Segmentation with Semantics

We propose to explore a new problem called audio-visual segmentation (AVS),
in which the goal is to output a pixel-level map of the object(s) that produce
sound at the time of the image frame. To facilitate this research, we construct
the first audio-visual segmentation benchmark (AVSBench), providing pixel-wise
annotations for the sounding objects in audible videos. Two settings are
studied with this benchmark: 1) semi-supervised audio-visual segmentation with
a single sound source and 2) fully-supervised audio-visual segmentation with
multiple sound sources. To deal with the AVS problem, we propose a novel method
that uses a temporal pixel-wise audio-visual interaction module to inject audio
semantics as guidance for the visual segmentation process. We also design a
regularization loss to encourage the audio-visual mapping during training.
Quantitative and qualitative experiments on the AVSBench compare our approach
to several existing methods from related tasks, demonstrating that the proposed
method is promising for building a bridge between the audio and pixel-wise
visual semantics. Code is available at this https URL

本文提出音频与视觉分割（AVS）问题，并通过构建 AVSBench 基准集进行了研究。通过引入一种新颖的方法 —— 时间上的像素级音视频交互模块，可以指导视觉分割过程，同时设计了正则化损失函数以鼓励音视频映射的训练，比较了几种现有方法，发现该方法有望在音频和像素级视觉语义之间搭建桥梁。