Recent advances in visually-induced audio generation are based on sampling
short, low-fidelity, and one-class sounds. Moreover, sampling 1 second of audio
from the state-of-the-art model takes minutes on a high-end GPU. In this work,
we propose a single model capable of generating visually relevant,
high-fidelity sounds prompted with a set of frames from open-domain videos in
less time than it takes to play it on a single GPU.
We train a transformer to sample a new spectrogram from the pre-trained
spectrogram codebook given the set of video features. The codebook is obtained
using a variant of VQGAN trained to produce a compact sampling space with a
novel spectrogram-based perceptual loss. The generated spectrogram is
transformed into a waveform using a window-based GAN that significantly speeds
up generation. Considering the lack of metrics for automatic evaluation of
generated spectrograms, we also build a family of metrics called FID and MKL.
These metrics are based on a novel sound classifier, called Melception, and
designed to evaluate the fidelity and relevance of open-domain samples.
Both qualitative and quantitative studies are conducted on small- and
large-scale datasets to evaluate the fidelity and relevance of generated
samples. We also compare our model to the state-of-the-art and observe a
substantial improvement in quality, size, and computation time. Code, demo, and
samples: v-iashin.github.io/SpecVQGAN

本文在视觉诱发音频生成方面进行了研究，提出了一种单一模型，可以在比单个 GPU 播放时间更短的时间内生成高保真、与视觉相符的声音，并使用新型的 FID 和 MKL 度量标准对其进行了评估。