The recent success of large pre-trained language models such as BERT and
GPT-2 has suggested the effectiveness of incorporating language priors in
downstream dialog generation tasks. However, the performance of pre-trained
models on the dialog task is not as optimal as expected. In this paper, we
propose a Pre-trained Role Alternating Language model (PRAL), designed
specifically for task-oriented conversational systems. We adopted (Wu et al.,
2019) that models two speakers separately. We also design several techniques,
such as start position randomization, knowledge distillation, and history
discount to improve pre-training performance. We introduce a task-oriented
dialog pretraining dataset by cleaning 13 existing data sets. We test PRAL on
three different downstream tasks. The results show that PRAL performs better or
on par with state-of-the-art methods.

本文提出一种专门为特定任务对话系统设计的预训练模型 PRAL，并设计采用多种技术来提高模型性能及引入了对话预训练数据集，实验结果表明 PRAL 在三种不同任务中表现优异，超越或与最先进的方法持平。