Despite the recent advances in applying pre-trained language models to
generate high-quality texts, generating long passages that maintain long-range
coherence is yet challenging for these models. In this paper, we propose
DiscoDVT, a discourse-aware discrete variational Transformer to tackle the
incoherence issue. DiscoDVT learns a discrete variable sequence that summarizes
the global structure of the text and then applies it to guide the generation
process at each decoding step. To further embed discourse-aware information
into the discrete latent representations, we introduce an auxiliary objective
to model the discourse relations within the text. We conduct extensive
experiments on two open story generation datasets and demonstrate that the
latent codes learn meaningful correspondence to the discourse structures that
guide the model to generate long texts with better long-range coherence.

本文提出 DiscoDVT：一种基于离散变分 Transformer 的话语感知模型，用于帮助预训练语言模型生成保持长距离连贯性的长文章。通过学习一个可以概括全文结构的离散变量序列，并利用其指导每个解码步骤的生成过程。试验表明，离散编码与话语结构有显著联系，提高长段连贯性。