We present SongComposer, an innovative LLM designed for song composition. It
could understand and generate melodies and lyrics in symbolic song
representations, by leveraging the capability of LLM. Existing music-related
LLM treated the music as quantized audio signals, while such implicit encoding
leads to inefficient encoding and poor flexibility. In contrast, we resort to
symbolic song representation, the mature and efficient way humans designed for
music, and enable LLM to explicitly compose songs like humans. In practice, we
design a novel tuple design to format lyric and three note attributes (pitch,
duration, and rest duration) in the melody, which guarantees the correct LLM
understanding of musical symbols and realizes precise alignment between lyrics
and melody. To impart basic music understanding to LLM, we carefully collected
SongCompose-PT, a large-scale song pretraining dataset that includes lyrics,
melodies, and paired lyrics-melodies in either Chinese or English. After
adequate pre-training, 10K carefully crafted QA pairs are used to empower the
LLM with the instruction-following capability and solve diverse tasks. With
extensive experiments, SongComposer demonstrates superior performance in
lyric-to-melody generation, melody-to-lyric generation, song continuation, and
text-to-song creation, outperforming advanced LLMs like GPT-4.

SongComposer 是一种创新的基于 LLM 的歌曲作曲技术，通过利用 LLM 的能力来理解和生成具有象征性歌曲表示的旋律和歌词，通过符号化歌曲表示的方式，使 LLM 能够像人类一样明确地作曲，通过 SongCompose-PT 预训练数据集的收集和充分的预训练，SongComposer 在歌词到旋律生成、旋律到歌词生成、歌曲延续和文本到歌曲创作等领域展现出优越的性能。

SongComposer: 歌曲生成中用于歌词和旋律创作的大型语言模型

SongComposer: A Large Language Model for Lyric and Melody Composition in  Song Generation

Lyric-to-melody generation is an important task in songwriting, and is also
quite challenging due to its unique characteristics: the generated melodies
should not only follow good musical patterns, but also align with features in
lyrics such as rhythms and structures. These characteristics cannot be well
handled by neural generation models that learn lyric-to-melody mapping in an
end-to-end way, due to several issues: (1) lack of aligned lyric-melody
training data to sufficiently learn lyric-melody feature alignment; (2) lack of
controllability in generation to better and explicitly align the lyric-melody
features. In this paper, we propose Re-creation of Creations (ROC), a new
paradigm for lyric-to-melody generation. ROC generates melodies according to
given lyrics and also conditions on user-designated chord progression. It
addresses the above issues through a generation-retrieval pipeline.
Specifically, our paradigm has two stages: (1) creation stage, where a huge
amount of music fragments generated by a neural melody language model are
indexed in a database through several key features (e.g., chords, tonality,
rhythm, and structural information); (2) re-creation stage, where melodies are
re-created by retrieving music fragments from the database according to the key
features from lyrics and concatenating best music fragments based on
composition guidelines and melody language model scores. ROC has several
advantages: (1) It only needs unpaired melody data to train melody language
model, instead of paired lyric-melody data in previous models. (2) It achieves
good lyric-melody feature alignment in lyric-to-melody generation. Tested by
English and Chinese lyrics, ROC outperforms previous neural based
lyric-to-melody generation models on both objective and subjective metrics.

本研究介绍了一种新的歌词旋律生成范例，即 Re-creation of Creations (ROC)，该范例通过将大量由神经旋律语言模型生成的音乐片段分阶段储存在数据库中，再根据作曲指南和旋律语言模型的重要性评分，检索音乐片段以从歌词和和弦中重新创作旋律，实现好的歌词旋律特征对齐。

创作的再创作：歌词转旋律生成的新范式

Re-creation of Creations: A New Paradigm for Lyric-to-Melody Generation

Automatic song writing aims to compose a song (lyric and/or melody) by
machine, which is an interesting topic in both academia and industry. In
automatic song writing, lyric-to-melody generation and melody-to-lyric
generation are two important tasks, both of which usually suffer from the
following challenges: 1) the paired lyric and melody data are limited, which
affects the generation quality of the two tasks, considering a lot of paired
training data are needed due to the weak correlation between lyric and melody;
2) Strict alignments are required between lyric and melody, which relies on
specific alignment modeling. In this paper, we propose SongMASS to address the
above challenges, which leverages masked sequence to sequence (MASS)
pre-training and attention based alignment modeling for lyric-to-melody and
melody-to-lyric generation. Specifically, 1) we extend the original
sentence-level MASS pre-training to song level to better capture long
contextual information in music, and use a separate encoder and decoder for
each modality (lyric or melody); 2) we leverage sentence-level attention mask
and token-level attention constraint during training to enhance the alignment
between lyric and melody. During inference, we use a dynamic programming
strategy to obtain the alignment between each word/syllable in lyric and note
in melody. We pre-train SongMASS on unpaired lyric and melody datasets, and
both objective and subjective evaluations demonstrate that SongMASS generates
lyric and melody with significantly better quality than the baseline method
without pre-training or alignment constraint.

本文提出了 SongMASS 方法来解决自动生成歌曲时歌词到旋律生成和旋律到歌词生成所面临的限制问题，使用基于掩蔽序列到序列预训练和基于注意力的对齐建模，表现出明显优于基线方法的歌词和旋律生成质量。