Recent advances in pre-trained language models have significantly improved neural response generation. However, existing methods usually view the dialogue context as a linear sequence of tokens and learn to generate the next word through token-level self-attention. Such token-level encoding hinders the exploration of discourse-level coherence among utterances. This paper presents DialogBERT, a novel conversational response generation model that enhances previous PLM-based dialogue models. DialogBERT employs a hierarchical Transformer architecture. To efficiently capture the discourse-level coherence among utterances, we propose two training objectives, including masked utterance regression and distributed utterance order ranking in analogy to the original BERT training. Experiments on three multi-turn conversation datasets show that our approach remarkably outperforms the baselines, such as BART and DialoGPT, in terms of quantitative evaluation. The human evaluation suggests that DialogBERT generates more coherent, informative, and human-like responses than the baselines with significant margins.

本文提出了 DialogBERT，这是一种新型的对话响应生成模型，通过使用分层Transformer架构和两个训练目标，包括掩码语调回归和分布式语调顺序排名来提高先前基于PLM的对话模型，在三个多轮对话数据集上进行实验证明，相比于BART和DialoGPT，DialogBERT在定量评估方面表现出色，并且在人类评估中证明，DialogBERT比基线模型生成的响应更连贯、信息量更大，并更加接近人类表现。

DialogBERT: 通过学习恢复和排序话语生成具有话语意识的响应