Automatic music generation with artificial intelligence typically requires a
large amount of data which is hard to obtain for many less common genres and
musical instruments. To tackle this issue, we present ongoing work and
preliminary findings on the possibility for deep models to transfer knowledge
from language to music, by finetuning large language models pre-trained on a
massive text corpus on only hundreds of MIDI files of drum performances. We
show that by doing so, one of the largest, state-of-the-art models (GPT3) is
capable of generating reasonable drum grooves, while models that are not
pre-trained (Transformer) shows no such ability beyond naive repetition.
Evaluating generated music is a challenging task, more so is evaluating drum
grooves with little precedence in literature. Hence, we propose a tailored
structural evaluation method and analyze drum grooves produced by GPT3 compared
to those played by human professionals, exposing the strengths and weaknesses
of such generation by language-to-music transfer. Our findings suggest that
language-to-music transfer learning with large language models is viable and
promising.

本研究提出使用深度模型进行跨领域的知识迁移，将大规模预训练的语言模型应用于鼓演奏的 MIDI 文件生成。研究表明，GPT3 模型能够生成相对合理的鼓声，但使用其他没有预训练的模型却无法实现。同时，作者还提出了定制结构评估方法，以比较生成的音乐与专业音乐家演奏的音乐，并揭示了此种方法的优缺点。该研究表明，使用大型语言模型进行跨领域迁移学习具有应用前景。