In this paper, we explore the application of Large Language Models (LLMs) to
the pre-training of music. While the prevalent use of MIDI in music modeling is
well-established, our findings suggest that LLMs are inherently more compatible
with ABC Notation, which aligns more closely with their design and strengths,
thereby enhancing the model's performance in musical composition. To address
the challenges associated with misaligned measures from different tracks during
generation, we propose the development of a \underline{S}ynchronized
\underline{M}ulti-\underline{T}rack ABC Notation (\textbf{SMT-ABC Notation}),
which aims to preserve coherence across multiple musical tracks. Our
contributions include a series of models capable of handling up to 8192 tokens,
covering 90\% of the symbolic music data in our training set. Furthermore, we
explore the implications of the \underline{S}ymbolic \underline{M}usic
\underline{S}caling Law (\textbf{SMS Law}) on model performance. The results
indicate a promising direction for future research in music generation,
offering extensive resources for community-led research through our open-source
contributions.

本文探讨了大型语言模型（LLMs）在音乐的预训练中的应用。研究发现，LLMs 与 ABC 记谱法更加兼容，提高了音乐创作模型的性能。为了解决不同音轨间的错位问题，我们提出了一种名为 Synchronized Multi-Track ABC Notation（SMT-ABC Notation）的方法，旨在保持多个音乐音轨之间的一致性。我们的贡献包括一系列能够处理 8192 个标记的模型，覆盖了训练集中 90％的符号音乐数据。此外，我们还研究了符号音乐缩放定律（SMS Law）对模型性能的影响。研究结果表明了音乐生成领域未来研究的一个有前途的方向，并通过我们的开源贡献为社区主导的研究提供了丰富的资源。

MuPT: 一个生成式的符号音乐预训练 Transformer

MuPT: A Generative Symbolic Music Pretrained Transformer

While Large Language Models (LLMs) demonstrate impressive capabilities in
text generation, we find that their ability has yet to be generalized to music,
humanity's creative language. We introduce ChatMusician, an open-source LLM
that integrates intrinsic musical abilities. It is based on continual
pre-training and finetuning LLaMA2 on a text-compatible music representation,
ABC notation, and the music is treated as a second language. ChatMusician can
understand and generate music with a pure text tokenizer without any external
multi-modal neural structures or tokenizers. Interestingly, endowing musical
abilities does not harm language abilities, even achieving a slightly higher
MMLU score. Our model is capable of composing well-structured, full-length
music, conditioned on texts, chords, melodies, motifs, musical forms, etc,
surpassing GPT-4 baseline. On our meticulously curated college-level music
understanding benchmark, MusicTheoryBench, ChatMusician surpasses LLaMA2 and
GPT-3.5 on zero-shot setting by a noticeable margin. Our work reveals that LLMs
can be an excellent compressor for music, but there remains significant
territory to be conquered. We release our 4B token music-language corpora
MusicPile, the collected MusicTheoryBench, code, model and demo in GitHub.

我们介绍了 ChatMusician，这是一个开源的大型语言模型，基于在 ABC 符号表示和音乐作为第二语言上进行连续预训练和微调的 LLaMA2，它能够理解和生成音乐，并能够创作结构良好、完整的音乐作品，超过了 GPT-4 基准模型。

ChatMusician：理解和生成音乐的 LLM 方法

ChatMusician: Understanding and Generating Music Intrinsically with LLM

We apply deep learning methods, specifically long short-term memory (LSTM)
networks, to music transcription modelling and composition. We build and train
LSTM networks using approximately 23,000 music transcriptions expressed with a
high-level vocabulary (ABC notation), and use them to generate new
transcriptions. Our practical aim is to create music transcription models
useful in particular contexts of music composition. We present results from
three perspectives: 1) at the population level, comparing descriptive
statistics of the set of training transcriptions and generated transcriptions;
2) at the individual level, examining how a generated transcription reflects
the conventions of a music practice in the training transcriptions (Celtic
folk); 3) at the application level, using the system for idea generation in
music composition. We make our datasets, software and sound examples open and
available: https://github.com/IraKorshunova/folk-rnn.

本文采用深度学习方法，具体来说是 LSTM 网络，用于音乐转录建模和创作，使用大约 23,000 个高级词汇（ABC 符号）表示的音乐转录来构建和训练 LSTM 网络，并用其生成新的转录。实际目的是在特定的音乐创作情境中创建有用的音乐转录模型，我们从三个方面展示了结果：1）在种群水平上，比较训练转录和生成转录集的描述性统计数据；2）在个体水平上，研究生成的转录如何反映训练转录中音乐实践的惯例（凯尔特民间音乐）；3）在应用级别上，使用该系统进行音乐创作的创意生成。我们开放和提供了我们的数据集、软件和声音示例：https://github.com/IraKorshunova/folk-rnn。