This paper presents a new Unified pre-trained Language Model (UniLM) that can
be fine-tuned for both natural language understanding and generation tasks. The
model is pre-trained using three types of language modeling tasks:
unidirectional, bidirectional, and sequence-to-sequence prediction. The unified
modeling is achieved by employing a shared Transformer network and utilizing
specific self-attention masks to control what context the prediction conditions
on. UniLM compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0
and CoQA question answering tasks. Moreover, UniLM achieves new
state-of-the-art results on five natural language generation datasets,
including improving the CNN/DailyMail abstractive summarization ROUGE-L to
40.51 (2.04 absolute improvement), the Gigaword abstractive summarization
ROUGE-L to 35.75 (0.86 absolute improvement), the CoQA generative question
answering F1 score to 82.5 (37.1 absolute improvement), the SQuAD question
generation BLEU-4 to 22.12 (3.75 absolute improvement), and the DSTC7
document-grounded dialog response generation NIST-4 to 2.67 (human performance
is 2.65). The code and pre-trained models are available at
this https URL

本文介绍了一种新的统一预训练语言模型 (UniLM)，该模型可用于自然语言理解和生成任务，使用了三种类型的语言建模任务进行预训练，采用共享 Transformer 网络和特定的自我注意掩码来控制预测条件的上下文，其在自然语言生成方面的表现优于 BERT，最终达到了五种自然语言生成数据集的最新最优成果。