Generating realistic audio for human interactions is important for many
applications, such as creating sound effects for films or virtual reality
games. Existing approaches implicitly assume total correspondence between the
video and audio during training, yet many sounds happen off-screen and have
weak to no correspondence with the visuals -- resulting in uncontrolled ambient
sounds or hallucinations at test time. We propose a novel ambient-aware audio
generation model, AV-LDM. We devise a novel audio-conditioning mechanism to
learn to disentangle foreground action sounds from the ambient background
sounds in in-the-wild training videos. Given a novel silent video, our model
uses retrieval-augmented generation to create audio that matches the visual
content both semantically and temporally. We train and evaluate our model on
two in-the-wild egocentric video datasets Ego4D and EPIC-KITCHENS. Our model
outperforms an array of existing methods, allows controllable generation of the
ambient sound, and even shows promise for generalizing to computer graphics
game clips. Overall, our work is the first to focus video-to-audio generation
faithfully on the observed visual content despite training from uncurated clips
with natural background sounds.

提出了一种新颖的环境感知音频生成模型，用于根据视频内容生成符合语义和时间要求的音频；使用了特殊的音频条件机制，以在野外训练视频中学习将前景动作声音与环境背景声音解耦合。