Our objective is to transform a video into a set of discrete audio-visual
objects using self-supervised learning. To this end, we introduce a model that
uses attention to localize and group sound sources, and optical flow to
aggregate information over time. We demonstrate the effectiveness of the
audio-visual object embeddings that our model learns by using them for four
downstream speech-oriented tasks: (a) multi-speaker sound source separation,
(b) localizing and tracking speakers, (c) correcting misaligned audio-visual
data, and (d) active speaker detection. Using our representation, these tasks
can be solved entirely by training on unlabeled video, without the aid of
object detectors. We also demonstrate the generality of our method by applying
it to non-human speakers, including cartoons and puppets.Our model
significantly outperforms other self-supervised approaches, and obtains
performance competitive with methods that use supervised face detection.

本文提出了一种使用自监督学习把视频转化成一组离散的音频 - 视觉物体的模型，并通过注意力定位和分组声源，光流聚合信息等方式提高了模型的准确度。实验表明，本模型学习到的音频 - 视觉对象嵌入可用于解决多说话者音源分离、说话人定位、音频 - 视觉数据校正和活跃说话人检测等四个任务，同时，使用非人类演讲者数据表明了本方法的可泛化性和比其他相关研究更好的效果。