Neural text generation has made tremendous progress in various tasks. One
common characteristic of most of the tasks is that the texts are not restricted
to some rigid formats when generating. However, we may confront some special
text paradigms such as Lyrics (assume the music score is given), Sonnet, SongCi
(classical Chinese poetry of the Song dynasty), etc. The typical
characteristics of these texts are in three folds: (1) They must comply fully
with the rigid predefined formats. (2) They must obey some rhyming schemes. (3)
Although they are restricted to some formats, the sentence integrity must be
guaranteed. To the best of our knowledge, text generation based on the
predefined rigid formats has not been well investigated. Therefore, we propose
a simple and elegant framework named SongNet to tackle this problem. The
backbone of the framework is a Transformer-based auto-regressive language
model. Sets of symbols are tailor-designed to improve the modeling performance
especially on format, rhyme, and sentence integrity. We improve the attention
mechanism to impel the model to capture some future information on the format.
A pre-training and fine-tuning framework is designed to further improve the
generation quality. Extensive experiments conducted on two collected corpora
demonstrate that our proposed framework generates significantly better results
in terms of both automatic metrics and the human evaluation.

SongNet 是一个基于 Transformer 的自回归语言模型，用于生成音乐歌词、十四行诗、宋词等格式严格的文本，相比于其他模型，在格式、韵律和句子完整性等方面都有较大改进。