Non-task oriented dialogue systems have achieved great success in recent
years due to largely accessible conversation data and the development of deep
learning techniques. Given a context, current systems are able to yield a
relevant and fluent response, but sometimes make logical mistakes because of
weak reasoning capabilities. To facilitate the conversation reasoning research,
we introduce MuTual, a novel dataset for Multi-Turn dialogue Reasoning,
consisting of 8,860 manually annotated dialogues based on Chinese student
English listening comprehension exams. Compared to previous benchmarks for
non-task oriented dialogue systems, MuTual is much more challenging since it
requires a model that can handle various reasoning problems. Empirical results
show that state-of-the-art methods only reach 71%, which is far behind the
human performance of 94%, indicating that there is ample room for improving
reasoning ability. MuTual is available at this https URL

介绍了一个新的多轮对话推理数据集 MuTual，包含 8,860 个中文学生英语听力考试基础上的手动注释对话，对话系统或模型需要具备处理各种推理问题的能力，现有的最先进的方法仅可达到 71％，远远落后于 94％的人类表现