Conventional autoregressive left-to-right (L2R) sequence generation faces two
issues during decoding: limited to unidirectional target sequence modeling, and
constrained on strong local dependencies. To address the aforementioned
problem, we propose P$^3$LM, a probabilistically permuted prophet language
model, which strengthens the modeling of bidirectional information and long
token dependencies for sequence generation. Specifically, P$^3$LM learns to
generate tokens in permuted order upon an order-aware transformer decoder, as
well as to generate the corresponding future $N$ tokens with a multi-stream
attention mechanism. Extensive experiments are conducted on the GLGE benchmark,
which includes four datasets for summarization, two for question generation,
one for conversational question answering, and one for dialog response
generation, where P$^3$LM achieves state-of-the-art results compared with
strong publicly available generative pre-training methods.

本文提出了一种名为 P3LM 的语言模型，通过学习生成排列顺序上的 token 和对应的未来 N 个 token，进一步加强了双向信息建模和长程相关性建模。在四个摘要数据集、两个问题生成数据集、一个对话问题回答数据集和一个对话回复数据集的广泛实验中，证明了 P3LM 在生成式预训练方法上具有最先进的结果。