Large models for text-to-music generation have achieved significant progress,
facilitating the creation of high-quality and varied musical compositions from
provided text prompts. However, input text prompts may not precisely capture
user requirements, particularly when the objective is to generate music that
embodies a specific concept derived from a designated reference collection. In
this paper, we propose a novel method for customized text-to-music generation,
which can capture the concept from a two-minute reference music and generate a
new piece of music conforming to the concept. We achieve this by fine-tuning a
pretrained text-to-music model using the reference music. However, directly
fine-tuning all parameters leads to overfitting issues. To address this
problem, we propose a Pivotal Parameters Tuning method that enables the model
to assimilate the new concept while preserving its original generative
capabilities. Additionally, we identify a potential concept conflict when
introducing multiple concepts into the pretrained model. We present a concept
enhancement strategy to distinguish multiple concepts, enabling the fine-tuned
model to generate music incorporating either individual or multiple concepts
simultaneously. Since we are the first to work on the customized music
generation task, we also introduce a new dataset and evaluation protocol for
the new task. Our proposed Jen1-DreamStyler outperforms several baselines in
both qualitative and quantitative evaluations. Demos will be available at
this https URL

通过精调预训练的文本到音乐模型，本论文提出了一种新的方法来进行定制化的文本到音乐生成，利用参考音乐中的概念生成符合该概念的新音乐，并通过关键参数调整解决过拟合问题，同时提出概念增强策略以区分多个概念，该方法在定性和定量评估中优于其他基线模型。

JEN-1 梦幻造型师：通过关键参数调整实现个性化音乐概念学习

JEN-1 DreamStyler: Customized Musical Concept Learning via Pivotal  Parameters Tuning

Text-to-music generation (T2M-Gen) faces a major obstacle due to the scarcity
of large-scale publicly available music datasets with natural language
captions. To address this, we propose the Music Understanding LLaMA (MU-LLaMA),
capable of answering music-related questions and generating captions for music
files. Our model utilizes audio representations from a pretrained MERT model to
extract music features. However, obtaining a suitable dataset for training the
MU-LLaMA model remains challenging, as existing publicly accessible audio
question answering datasets lack the necessary depth for open-ended music
question answering. To fill this gap, we present a methodology for generating
question-answer pairs from existing audio captioning datasets and introduce the
MusicQA Dataset designed for answering open-ended music-related questions. The
experiments demonstrate that the proposed MU-LLaMA model, trained on our
designed MusicQA dataset, achieves outstanding performance in both music
question answering and music caption generation across various metrics,
outperforming current state-of-the-art (SOTA) models in both fields and
offering a promising advancement in the T2M-Gen research field.

文中提出的 MU-LLaMA 模型通过 MusicQA 数据集的训练，在音乐问答和音乐配音生成方面取得了出色的表现，超越了当前领先的模型，并为 T2M-Gen 研究领域带来了有希望的进展。