In recent times, the focus on text-to-audio (TTA) generation has intensified,
as researchers strive to synthesize audio from textual descriptions. However,
most existing methods, though leveraging latent diffusion models to learn the
correlation between audio and text embeddings, fall short when it comes to
maintaining a seamless synchronization between the produced audio and its
video. This often results in discernible audio-visual mismatches. To bridge
this gap, we introduce a groundbreaking benchmark for Text-to-Audio generation
that aligns with Videos, named T2AV-Bench. This benchmark distinguishes itself
with three novel metrics dedicated to evaluating visual alignment and temporal
consistency. To complement this, we also present a simple yet effective
video-aligned TTA generation model, namely T2AV. Moving beyond traditional
methods, T2AV refines the latent diffusion approach by integrating
visual-aligned text embeddings as its conditional foundation. It employs a
temporal multi-head attention transformer to extract and understand temporal
nuances from video data, a feat amplified by our Audio-Visual ControlNet that
adeptly merges temporal visual representations with text embeddings. Further
enhancing this integration, we weave in a contrastive learning objective,
designed to ensure that the visual-aligned text embeddings resonate closely
with the audio features. Extensive evaluations on the AudioCaps and T2AV-Bench
demonstrate that our T2AV sets a new standard for video-aligned TTA generation
in ensuring visual alignment and temporal consistency.

近期，研究人员在文本转语音生成方面的关注度不断增加，本研究提出了一个名为 T2AV-Bench 的创新基准和一个简单而有效的视频对齐文本到音频生成模型 T2AV，通过整合视觉对齐的文本嵌入到生成模型中，通过时间多头注意力转换器从视频数据中提取和理解时间细微差异，并通过 Audio-Visual ControlNet 将时态视觉表示与文本嵌入精确地融合，进一步增强集成性，以确保视觉对齐和时间一致性。经过对 AudioCaps 和 T2AV-Bench 的广泛评估，T2AV 在视觉对齐和时间一致性上设立了新的标准。