Recent large-scale neural autoregressive sequence models have shown
impressive performances on a variety of natural language generation tasks.
However, their generated sequences often exhibit degenerate properties such as
non-termination, undesirable repetition, and premature termination, when
generated with decoding algorithms such as greedy search, beam search, top-$k$
sampling, and nucleus sampling. In this paper, we focus on the problem of
non-terminating sequences resulting from an incomplete decoding algorithm. We
first define an incomplete probable decoding algorithm which includes greedy
search, top-$k$ sampling, and nucleus sampling, beyond the incomplete decoding
algorithm originally put forward by Welleck et al. (2020). We then propose a
non-monotonic self-terminating language model, which significantly relaxes the
constraint of monotonically increasing termination probability in the
originally proposed self-terminating language model by Welleck et al. (2020),
to address the issue of non-terminating sequences when using incomplete
probable decoding algorithms. We prove that our proposed model prevents
non-terminating sequences when using not only incomplete probable decoding
algorithms but also beam search. We empirically validate our model on sequence
completion tasks with various architectures.

本文研究大规模神经自回归序列模型的生成算法在 greedy search、beam search、top-k sampling 和 nucleus sampling 的情况下呈现的非终止、重复和过早中止等退化属性，提出了一个非单调自动终止语言模型，该模型通过放宽自动终止概率单调递增的限制来解决该问题，并证明了该模型不仅能防止非终止序列，而且还能在采用 beam search 等其他算法时使用。