Pre-trained conversation models (PCMs) have demonstrated remarkable results
in task-oriented dialogue (TOD) systems. Many PCMs focus predominantly on
dialogue management tasks like dialogue state tracking, dialogue generation
tasks like response generation, or both. However, the existing PCMs seldom
consider dialogue comprehension tasks, such as dialogue question answering and
summarization tasks. These tasks allow PCMs to glean dialogue context from
various angles. This observation naturally raises the question: Can the
performance of downstream dialogue tasks be enhanced if a PCM is pre-trained on
dialogue management, generation, and comprehension tasks?
To investigate this, we proposed an Omnipotent Dialogue pre-training model
(OmniDialog). It unifies these three dialogue tasks into a monolithic framework
by multi-task learning, fostering inter-task communication. The pre-training
corpus of OmniDialog spans $\mathbf{7}$ dialogue-focused tasks, drawing from
$\mathbf{15}$ datasets and encompassing over $\mathbf{3.2}$ million dialogue
utterances. To our knowledge, OmniDialog is a pioneering PCM pre-trained across
dialogue management, generation, and comprehension domains. We evaluated its
performance across four tasks: dialogue summarization, end-to-end dialogue
modeling, dialogue state tracking, and intent classification. The results
underscore its efficacy in domain transfer learning, low-resource, and
full-dataset scenarios. Furthermore, to glean a nuanced understanding of
OmniDialog's strengths and potential pitfalls, we designed a fine-grained
analysis framework for dialogue-centric tasks. Experimental results show that
the OmniDialog is good at hard samples, such as long dialogues and lengthy
responses.

探讨了将 PCM 预训练于对话管理、生成和理解任务中的 OmniDialog 模型在对话任务中的效能。

OmniDialog: 用于任务导向对话系统的全能预训练模型

OmniDialog: An Omnipotent Pre-training Model for Task-Oriented Dialogue  System

Pre-trained conversation models (PCMs) have achieved promising progress in
recent years. However, existing PCMs for Task-oriented dialog (TOD) are
insufficient for capturing the sequential nature of the TOD-related tasks, as
well as for learning dialog policy information. To alleviate these problems,
this paper proposes a task-progressive PCM with two policy-aware pre-training
tasks. The model is pre-trained through three stages where TOD-related tasks
are progressively employed according to the task logic of the TOD system. A
global policy consistency task is designed to capture the multi-turn dialog
policy sequential relation, and an act-based contrastive learning task is
designed to capture similarities among samples with the same dialog policy. Our
model achieves better results on both MultiWOZ and In-Car end-to-end dialog
modeling benchmarks with only 18\% parameters and 25\% pre-training data
compared to the previous state-of-the-art PCM, GALAXY.

本论文提出了一种任务递进的预训练对话模型，通过两个具有策略感知的预训练任务来解决任务导向型对话模型中的时序性问题和学习对话策略信息的不足。该模型通过三个阶段的逐步任务处理来进行预训练，其中设计了一个全局策略一致性任务用于捕捉多轮对话策略的时序关系，并设计了一个基于行为的对比学习任务用于捕捉相同对话策略样本的相似性。与之前最先进的 PCM（GALAXY）相比，我们的模型仅使用 18％的参数和 25％的预训练数据，在 MultiWOZ 和 In-Car 的端到端对话建模基准上取得了更好的结果。