Lyrics-to-melody generation is an interesting and challenging topic in AI
music research field. Due to the difficulty of learning the correlations
between lyrics and melody, previous methods suffer from low generation quality
and lack of controllability. Controllability of generative models enables human
interaction with models to generate desired contents, which is especially
important in music generation tasks towards human-centered AI that can
facilitate musicians in creative activities. To address these issues, we
propose a controllable lyrics-to-melody generation network, ConL2M, which is
able to generate realistic melodies from lyrics in user-desired musical style.
Our work contains three main novelties: 1) To model the dependencies of music
attributes cross multiple sequences, inter-branch memory fusion (Memofu) is
proposed to enable information flow between multi-branch stacked LSTM
architecture; 2) Reference style embedding (RSE) is proposed to improve the
quality of generation as well as control the musical style of generated
melodies; 3) Sequence-level statistical loss (SeqLoss) is proposed to help the
model learn sequence-level features of melodies given lyrics. Verified by
evaluation metrics for music quality and controllability, initial study of
controllable lyrics-to-melody generation shows better generation quality and
the feasibility of interacting with users to generate the melodies in desired
musical styles when given lyrics.

我们提出了一种可控的歌词到旋律生成网络，能够以用户期望的音乐风格从歌词中生成逼真的旋律，通过验证各项指标，初始控制生成的旋律的研究表明具有更好的生成质量和与用户的互动性。