Pre-trained language models (PrLMs) have demonstrated superior performance
due to their strong ability to learn universal language representations from
self-supervised pre-training. However, even with the help of the powerful
PrLMs, it is still challenging to effectively capture task-related knowledge
from dialogue texts which are enriched by correlations among speaker-aware
utterances. In this work, we present SPIDER, Structural Pre-traIned DialoguE
Reader, to capture dialogue exclusive features. To simulate the dialogue-like
features, we propose two training objectives in addition to the original LM
objectives: 1) utterance order restoration, which predicts the order of the
permuted utterances in dialogue context; 2) sentence backbone regularization,
which regularizes the model to improve the factual correctness of summarized
subject-verb-object triplets. Experimental results on widely used dialogue
benchmarks verify the effectiveness of the newly introduced self-supervised
tasks.

本文提出了结构化预训练对话阅读器 SPIDER，通过两个训练目标相结合以捕获对话独有的特征，即复原语音顺序和句子骨干规则化，实现了从对话文本中有效地捕获任务相关知识，并在广泛使用的对话基准测试中取得了显著的实验结果。