This paper introduces FALL-E, a foley synthesis system and its
training/inference strategies. The FALL-E model employs a cascaded approach
comprising low-resolution spectrogram generation, spectrogram super-resolution,
and a vocoder. We trained every sound-related model from scratch using our
extensive datasets, and utilized a pre-trained language model. We conditioned
the model with dataset-specific texts, enabling it to learn sound quality and
recording environment based on text input. Moreover, we leveraged external
language models to improve text descriptions of our datasets and performed
prompt engineering for quality, coherence, and diversity. FALL-E was evaluated
by an objective measure as well as listening tests in the DCASE 2023 challenge
Task 7. The submission achieved the second place on average, while achieving
the best score for diversity, second place for audio quality, and third place
for class fitness.

本文介绍了 FALL-E—— 一个 foley 合成系统及其训练 / 推理策略，FALL-E 采用级联方法，由低分辨率谱图生成、谱图超分辨率和声码器组成，利用数据集训练每个与声音相关的模型，并利用预训练的语言模型，通过数据集特定的文本对模型进行条件设定。在 DCASE 2023 挑战任务 7 的客观测试和听辨测试中，FALL-E 取得了第二名的好成绩，在多样性得分上排名第一，在音质上排名第二，在分类适应度上排名第三。

FALL-E: 佛利音效合成模型与策略

FALL-E: A Foley Sound Synthesis Model and Strategies

"Foley" refers to sound effects that are added to multimedia during
post-production to enhance its perceived acoustic properties, e.g., by
simulating the sounds of footsteps, ambient environmental sounds, or visible
objects on the screen. While foley is traditionally produced by foley artists,
there is increasing interest in automatic or machine-assisted techniques
building upon recent advances in sound synthesis and generative models. To
foster more participation in this growing research area, we propose a challenge
for automatic foley synthesis. Through case studies on successful previous
challenges in audio and machine learning, we set the goals of the proposed
challenge: rigorous, unified, and efficient evaluation of different foley
synthesis systems, with an overarching goal of drawing active participation
from the research community. We outline the details and design considerations
of a foley sound synthesis challenge, including task definition, dataset
requirements, and evaluation criteria.

本文旨在探索利用机器辅助技术进行 Foley 音效合成的新方法并提出了一个详细的 Foley 合成挑战，包括任务定义、数据集需求和评估标准。