Consistency Identification has obtained remarkable success on open-domain
dialogue, which can be used for preventing inconsistent response generation.
However, in contrast to the rapid development in open-domain dialogue, few
efforts have been made to the task-oriented dialogue direction. In this paper,
we argue that consistency problem is more urgent in task-oriented domain. To
facilitate the research, we introduce CI-ToD, a novel dataset for Consistency
Identification in Task-oriented Dialog system. In addition, we not only
annotate the single label to enable the model to judge whether the system
response is contradictory, but also provide more fine-grained labels (i.e.,
Dialogue History Inconsistency, User Query Inconsistency and Knowledge Base
Inconsistency) to encourage model to know what inconsistent sources lead to it.
Empirical results show that state-of-the-art methods only achieve 51.3%, which
is far behind the human performance of 93.2%, indicating that there is ample
room for improving consistency identification ability. Finally, we conduct
exhaustive experiments and qualitative analysis to comprehend key challenges
and provide guidance for future directions. All datasets and models are
publicly available at https://github.com/yizhen20133868/CI-ToD.

本文介绍了 CI-ToD 数据集，并提供了单标记和细粒度标记两种方法并进行数据注释以解决任务导向对话系统中的一致性预测问题，实验结果表明目前最先进的方法只能达到 51.3%，存在改进的空间。