This paper introduces FALL-E, a foley synthesis system and its
training/inference strategies. The FALL-E model employs a cascaded approach
comprising low-resolution spectrogram generation, spectrogram super-resolution,
and a vocoder. We trained every sound-related model from scratch using our
extensive datasets, and utilized a pre-trained language model. We conditioned
the model with dataset-specific texts, enabling it to learn sound quality and
recording environment based on text input. Moreover, we leveraged external
language models to improve text descriptions of our datasets and performed
prompt engineering for quality, coherence, and diversity. FALL-E was evaluated
by an objective measure as well as listening tests in the DCASE 2023 challenge
Task 7. The submission achieved the second place on average, while achieving
the best score for diversity, second place for audio quality, and third place
for class fitness.

本文介绍了 FALL-E—— 一个 foley 合成系统及其训练 / 推理策略，FALL-E 采用级联方法，由低分辨率谱图生成、谱图超分辨率和声码器组成，利用数据集训练每个与声音相关的模型，并利用预训练的语言模型，通过数据集特定的文本对模型进行条件设定。在 DCASE 2023 挑战任务 7 的客观测试和听辨测试中，FALL-E 取得了第二名的好成绩，在多样性得分上排名第一，在音质上排名第二，在分类适应度上排名第三。