We focus on the task of generating sound from natural videos, and the sound
should be both temporally and content-wise aligned with visual signals. This
task is extremely challenging because some sounds generated \emph{outside} a
camera can not be inferred from video content. The model may be forced to learn
an incorrect mapping between visual content and these irrelevant sounds. To
address this challenge, we propose a framework named REGNET. In this framework,
we first extract appearance and motion features from video frames to better
distinguish the object that emits sound from complex background information. We
then introduce an innovative audio forwarding regularizer that directly
considers the real sound as input and outputs bottlenecked sound features.
Using both visual and bottlenecked sound features for sound prediction during
training provides stronger supervision for the sound prediction. The audio
forwarding regularizer can control the irrelevant sound component and thus
prevent the model from learning an incorrect mapping between video frames and
sound emitted by the object that is out of the screen. During testing, the
audio forwarding regularizer is removed to ensure that REGNET can produce
purely aligned sound only from visual features. Extensive evaluations based on
Amazon Mechanical Turk demonstrate that our method significantly improves both
temporal and content-wise alignment. Remarkably, our generated sound can fool
the human with a 68.12% success rate. Code and pre-trained models are publicly
available at this https URL

本文提出了一个名为 REGNET 的框架，利用外观和运动特征从视频帧中提取物体发出的声音，结合创新的音频转发正则化器进行更强的声音预测监督，在训练过程中使用了可控制影响因素的正则化器，并通过 Amazon Mechanical Turk 进行了大量评估，成功实现了音频和视频的时序和内容对齐。