We consider the task of generating diverse and realistic videos guided by
natural audio samples from a wide variety of semantic classes. For this task,
the videos are required to be aligned both globally and temporally with the
input audio: globally, the input audio is semantically associated with the
entire output video, and temporally, each segment of the input audio is
associated with a corresponding segment of that video. We utilize an existing
text-conditioned video generation model and a pre-trained audio encoder model.
The proposed method is based on a lightweight adaptor network, which learns to
map the audio-based representation to the input representation expected by the
text-to-video generation model. As such, it also enables video generation
conditioned on text, audio, and, for the first time as far as we can ascertain,
on both text and audio. We validate our method extensively on three datasets
demonstrating significant semantic diversity of audio-video samples and further
propose a novel evaluation metric (AV-Align) to assess the alignment of
generated videos with input audio samples. AV-Align is based on the detection
and comparison of energy peaks in both modalities. In comparison to recent
state-of-the-art approaches, our method generates videos that are better
aligned with the input sound, both with respect to content and temporal axis.
We also show that videos produced by our method present higher visual quality
and are more diverse.

生成多样化和逼真的视频，根据语义类别广泛的自然音频样本进行引导。采用了基于轻量级适配器网络的方法，将音频基础表示映射到文本 - 视频生成模型所期望的输入表示，实现了对文本、音频以及文本和音频的生成视频。在三个数据集上验证了该方法，展示了生成的音频视频样本的显著语义多样性，并提出了一种新的评估度量（AV-Align）以评估生成视频与输入音频样本的对齐性。与最新的先进方法相比，我们的方法生成的视频在内容和时间轴上都与输入音频更好地对齐，并且呈现更高的视觉质量和多样性。