The Text to Audible-Video Generation (TAVG) task involves generating videos
with accompanying audio based on text descriptions. Achieving this requires
skillful alignment of both audio and video elements. To support research in
this field, we have developed a comprehensive Text to Audible-Video Generation
Benchmark (TAVGBench), which contains over 1.7 million clips with a total
duration of 11.8 thousand hours. We propose an automatic annotation pipeline to
ensure each audible video has detailed descriptions for both its audio and
video contents. We also introduce the Audio-Visual Harmoni score (AVHScore) to
provide a quantitative measure of the alignment between the generated audio and
video modalities. Additionally, we present a baseline model for TAVG called
TAVDiffusion, which uses a two-stream latent diffusion model to provide a
fundamental starting point for further research in this area. We achieve the
alignment of audio and video by employing cross-attention and contrastive
learning. Through extensive experiments and evaluations on TAVGBench, we
demonstrate the effectiveness of our proposed model under both conventional
metrics and our proposed metrics.

提出了一个用于生成文字描述音频视频的基准测试集（TAVGBench），包含超过 1.7 百万个剪辑，总时长达到 11.8 千小时。通过自动注释流程，确保每个音频视频都具有音频和视频内容的详细描述。引入了 Audio-Visual Harmoni 分数（AVHScore），用于定量衡量生成音频和视频之间的对齐程度。此外，还提出了名为 TAVDiffusion 的 TAVG 基线模型，使用双流潜在扩散模型提供深入研究该领域的基础起点。通过在 TAVGBench 上进行广泛实验和评估，展示了我们所提出模型在传统指标和我们提出的指标下的有效性。