Automatic song writing aims to compose a song (lyric and/or melody) by
machine, which is an interesting topic in both academia and industry. In
automatic song writing, lyric-to-melody generation and melody-to-lyric
generation are two important tasks, both of which usually suffer from the
following challenges: 1) the paired lyric and melody data are limited, which
affects the generation quality of the two tasks, considering a lot of paired
training data are needed due to the weak correlation between lyric and melody;
2) Strict alignments are required between lyric and melody, which relies on
specific alignment modeling. In this paper, we propose SongMASS to address the
above challenges, which leverages masked sequence to sequence (MASS)
pre-training and attention based alignment modeling for lyric-to-melody and
melody-to-lyric generation. Specifically, 1) we extend the original
sentence-level MASS pre-training to song level to better capture long
contextual information in music, and use a separate encoder and decoder for
each modality (lyric or melody); 2) we leverage sentence-level attention mask
and token-level attention constraint during training to enhance the alignment
between lyric and melody. During inference, we use a dynamic programming
strategy to obtain the alignment between each word/syllable in lyric and note
in melody. We pre-train SongMASS on unpaired lyric and melody datasets, and
both objective and subjective evaluations demonstrate that SongMASS generates
lyric and melody with significantly better quality than the baseline method
without pre-training or alignment constraint.

本文提出了 SongMASS 方法来解决自动生成歌曲时歌词到旋律生成和旋律到歌词生成所面临的限制问题，使用基于掩蔽序列到序列预训练和基于注意力的对齐建模，表现出明显优于基线方法的歌词和旋律生成质量。