Foley audio, critical for enhancing the immersive experience in multimedia
content, faces significant challenges in the AI-generated content (AIGC)
landscape. Despite advancements in AIGC technologies for text and image
generation, the foley audio dubbing remains rudimentary due to difficulties in
cross-modal scene matching and content correlation. Current text-to-audio
technology, which relies on detailed and acoustically relevant textual
descriptions, falls short in practical video dubbing applications. Existing
datasets like AudioSet, AudioCaps, Clotho, Sound-of-Story, and WavCaps do not
fully meet the requirements for real-world foley audio dubbing task. To address
this, we introduce the Multi-modal Image and Narrative Text Dubbing Dataset
(MINT), designed to enhance mainstream dubbing tasks such as literary story
audiobooks dubbing, image/silent video dubbing. Besides, to address the
limitations of existing TTA technology in understanding and planning complex
prompts, a Foley Audio Content Planning, Generation, and Alignment (CPGA)
framework is proposed, which includes a content planning module leveraging
large language models for complex multi-modal prompts comprehension.
Additionally, the training process is optimized using Proximal Policy
Optimization based reinforcement learning, significantly improving the
alignment and auditory realism of generated foley audio. Experimental results
demonstrate that our approach significantly advances the field of foley audio
dubbing, providing robust solutions for the challenges of multi-modal dubbing.
Even when utilizing the relatively lightweight GPT-2 model, our framework
outperforms open-source multimodal large models such as LLaVA, DeepSeek-VL, and
Moondream2. The dataset is available at this https URL .

此研究论文介绍了 AI 生成内容中的 foley 音频在多媒体内容中提升沉浸式体验的重要性和挑战，探讨了当前文本到音频技术在视频配音应用中的局限，并提出了多模态音频配音数据集（MINT）和一种基于大型语言模型的含有复合模态提示的内容规划、生成和对齐框架，通过增强学习方法优化训练过程，提高配音质量和生成音频的真实感，为多模态配音领域提供了强大的解决方案。