We introduce AudioScopeV2, a state-of-the-art universal audio-visual
on-screen sound separation system which is capable of learning to separate
sounds and associate them with on-screen objects by looking at in-the-wild
videos. We identify several limitations of previous work on audio-visual
on-screen sound separation, including the coarse resolution of spatio-temporal
attention, poor convergence of the audio separation model, limited variety in
training and evaluation data, and failure to account for the trade off between
preservation of on-screen sounds and suppression of off-screen sounds. We
provide solutions to all of these issues. Our proposed cross-modal and
self-attention network architectures capture audio-visual dependencies at a
finer resolution over time, and we also propose efficient separable variants
that are capable of scaling to longer videos without sacrificing much
performance. We also find that pre-training the separation model only on audio
greatly improves results. For training and evaluation, we collected new human
annotations of onscreen sounds from a large database of in-the-wild videos
(YFCC100M). This new dataset is more diverse and challenging. Finally, we
propose a calibration procedure that allows exact tuning of on-screen
reconstruction versus off-screen suppression, which greatly simplifies
comparing performance between models with different operating points. Overall,
our experimental results show marked improvements in on-screen separation
performance under much more general conditions than previous methods with
minimal additional computational complexity.

AudioScopeV2 是一个最先进的通用音频视觉屏幕上声音分离系统，能够通过观察野外视频来学习分离声音并将它们与屏幕上的对象关联起来，并提出了解决先前工作的若干限制的解决方案，并通过新的数据集进行评估，最终取得了显着的改进。

AudioScopeV2：音视频注意力架构的开放域屏幕声音分离校准

AudioScopeV2: Audio-Visual Attention Architectures for Calibrated  Open-Domain On-Screen Sound Separation

We present the Yahoo Flickr Creative Commons 100 Million Dataset (YFCC100M),
the largest public multimedia collection that has ever been released. The
dataset contains a total of 100 million media objects, of which approximately
99.2 million are photos and 0.8 million are videos, all of which carry a
Creative Commons license. Each media object in the dataset is represented by
several pieces of metadata, e.g. Flickr identifier, owner name, camera, title,
tags, geo, media source. The collection provides a comprehensive snapshot of
how photos and videos were taken, described, and shared over the years, from
the inception of Flickr in 2004 until early 2014. In this article we explain
the rationale behind its creation, as well as the implications the dataset has
for science, research, engineering, and development. We further present several
new challenges in multimedia research that can now be expanded upon with our
dataset.

该研究介绍了雅虎 Flickr 创意共享千万数据集 (YFCC100M)，其包含约 1 亿个多媒体对象，其中约 99.2 万个为照片，0.8 万个为视频，所有对象均带有创意共享许可证。该数据集提供了详细的元数据信息和从 2004 年到 2014 年初的照片和视频的共享情况，对于科学、研究、工程和开发等领域具有重要意义。