Multi-modality perception is essential to develop interactive intelligence.
In this work, we consider a new task of visual information-infused audio
inpainting, \ie synthesizing missing audio segments that correspond to their
accompanying videos. We identify two key aspects for a successful inpainter:
(1) It is desirable to operate on spectrograms instead of raw audios. Recent
advances in deep semantic image inpainting could be leveraged to go beyond the
limitations of traditional audio inpainting. (2) To synthesize visually
indicated audio, a visual-audio joint feature space needs to be learned with
synchronization of audio and video. To facilitate a large-scale study, we
collect a new multi-modality instrument-playing dataset called MUSIC-Extra-Solo
(MUSICES) by enriching MUSIC dataset. Extensive experiments demonstrate that
our framework is capable of inpainting realistic and varying audio segments
with or without visual contexts. More importantly, our synthesized audio
segments are coherent with their video counterparts, showing the effectiveness
of our proposed Vision-Infused Audio Inpainter (VIAI). Code, models, dataset
and video results are available at
this https URL

本文提出了一种名为视觉信息注入音频修补的多模态感知任务，通过学习视觉 - 音频联合特征空间，并在语音谱图上进行操作，能够修补音频中的缺失部分，实现音频与视频同步，经过大规模实验验证该方法的有效性。