Multi-modal learning in the audio-language domain has seen significant
advancements in recent years. However, audio-language learning faces challenges
due to limited and lower-quality data compared to image-language tasks.
Existing audio-language datasets are notably smaller, and manual labeling is
hindered by the need to listen to entire audio clips for accurate labeling.
Our method systematically generates audio-caption pairs by augmenting audio
clips with natural language labels and corresponding audio signal processing
operations. Leveraging a Large Language Model, we generate descriptions of
augmented audio clips with a prompt template. This scalable method produces
AudioSetMix, a high-quality training dataset for text-and-audio related models.
Integration of our dataset improves models performance on benchmarks by
providing diversified and better-aligned examples. Notably, our dataset
addresses the absence of modifiers (adjectives and adverbs) in existing
datasets. By enabling models to learn these concepts, and generating hard
negative examples during training, we achieve state-of-the-art performance on
multiple benchmarks.

我们通过增加自然语言标签和相应的音频信号处理操作，使用大型语言模型提供了一个高质量的训练数据集，该数据集在文本和音频相关模型的基准测试中提供了多样化且更好对齐的示例，从而改善了模型的性能。

AudioSetMix: 用 LLM 辅助增强音频 - 语言数据集

AudioSetMix: Enhancing Audio-Language Datasets with LLM-Assisted  Augmentations

We introduce the Song Describer dataset (SDD), a new crowdsourced corpus of
high-quality audio-caption pairs, designed for the evaluation of
music-and-language models. The dataset consists of 1.1k human-written natural
language descriptions of 706 music recordings, all publicly accessible and
released under Creative Common licenses. To showcase the use of our dataset, we
benchmark popular models on three key music-and-language tasks (music
captioning, text-to-music generation and music-language retrieval). Our
experiments highlight the importance of cross-dataset evaluation and offer
insights into how researchers can use SDD to gain a broader understanding of
model performance.

我们介绍了 “歌曲描述者数据集 (SDD)”，这是一个新的众包语料库，用于评估音乐和语言模型。该数据集包含 1.1k 个人工编写的自然语言描述和 706 个音乐录音的对应，全部都是公开可访问的，且都在创作共用许可证下发布。为了展示我们数据集的用途，我们在三个关键的音乐和语言任务上对一些热门模型进行了基准测试（音乐字幕生成、文本到音乐生成和音乐语言检索）。我们的实验强调了跨数据集评估的重要性，并提供了研究人员如何利用 SDD 来获得更全面的模型性能理解的见解。