Pre-trained language models have achieved impressive results in various music
understanding and generation tasks. However, existing pre-training methods for
symbolic melody generation struggle to capture multi-scale, multi-dimensional
structural information in note sequences, due to the domain knowledge
discrepancy between text and music. Moreover, the lack of available large-scale
symbolic melody datasets limits the pre-training improvement. In this paper, we
propose MelodyGLM, a multi-task pre-training framework for generating melodies
with long-term structure. We design the melodic n-gram and long span sampling
strategies to create local and global blank infilling tasks for modeling the
local and global structures in melodies. Specifically, we incorporate pitch
n-grams, rhythm n-grams, and their combined n-grams into the melodic n-gram
blank infilling tasks for modeling the multi-dimensional structures in
melodies. To this end, we have constructed a large-scale symbolic melody
dataset, MelodyNet, containing more than 0.4 million melody pieces. MelodyNet
is utilized for large-scale pre-training and domain-specific n-gram lexicon
construction. Both subjective and objective evaluations demonstrate that
MelodyGLM surpasses the standard and previous pre-training methods. In
particular, subjective evaluations show that, on the melody continuation task,
MelodyGLM achieves average improvements of 0.82, 0.87, 0.78, and 0.94 in
consistency, rhythmicity, structure, and overall quality, respectively.
Notably, MelodyGLM nearly matches the quality of human-composed melodies on the
melody inpainting task.

该研究提出了 MelodyGLM，一个多任务预训练框架，用于生成带有长期结构的旋律。研究使用 melodic n-gram 和长跨度抽样策略来创建本地和全局填充任务，以建模旋律中的本地和全局结构。研究还构建了一个包含超过 40 万旋律片段的大规模符号旋律数据集，用于大规模预训练和领域特定的 n-gram 词典构建。主观和客观评估结果表明，MelodyGLM 超过了标准和之前的预训练方法，在旋律连续性、节奏性、结构性和整体质量方面都取得了显著改进。值得注意的是，MelodyGLM 在旋律填充任务上几乎与人类创作的旋律的质量相媲美。