Generating realistic audio effects for movies and other media is a
challenging task that is accomplished today primarily through physical
techniques known as Foley art. Foley artists create sounds with common objects
(e.g., boxing gloves, broken glass) in time with video as it is playing to
generate captivating audio tracks. In this work, we aim to develop a
deep-learning based framework that does much the same - observes video in it's
natural sequence and generates realistic audio to accompany it. Notably, we
have reason to believe this is achievable due to advancements in realistic
audio generation techniques conditioned on other inputs (e.g., Wavenet
conditioned on text). We explore several different model architectures to
accomplish this task that process both previously-generated audio and video
context. These include deep-fusion CNN, dilated Wavenet CNN with visual
context, and transformer-based architectures. We find that the
transformer-based architecture yields the most promising results, matching
low-frequencies to visual patterns effectively, but failing to generate more
nuanced waveforms.

通过深度学习的框架及改进的人工音频生成技术，本研究旨在生成与视频相符的逼真音频效果。尝试了多种模型结构，其中基于 Transformer 的架构效果最为优秀，能有效地匹配低频的视觉模式，但在生成细微波形方面表现不佳。