Knowledge-based, open-domain dialogue generation aims to build chit-chat systems that talk to humans using mined support knowledge. Many types and sources of knowledge have previously been shown to be useful as support knowledge. Even in the era of large language models, response generation grounded in knowledge retrieved from additional up-to-date sources remains a practically important approach. While prior work using single-source knowledge has shown a clear positive correlation between the performances of knowledge selection and response generation, there are no existing multi-source datasets for evaluating support knowledge retrieval. Further, prior work has assumed that the knowledge sources available at test time are the same as during training. This unrealistic assumption unnecessarily handicaps models, as new knowledge sources can become available after a model is trained. In this paper, we present a high-quality benchmark named multi-source Wizard of Wikipedia (Ms.WoW) for evaluating multi-source dialogue knowledge selection and response generation. Unlike existing datasets, it contains clean support knowledge, grounded at the utterance level and partitioned into multiple knowledge sources. We further propose a new challenge, dialogue knowledge plug-and-play, which aims to test an already trained dialogue model on using new support knowledge from previously unseen sources in a zero-shot fashion.

构建基于知识的开放领域对话生成技术旨在利用挖掘的知识与人类进行闲聊。本文提出了一个高质量的多源对话知识选择和生成评估基准，名为multi-source Wizard of Wikipedia（Ms.WoW），用于评估多源对话知识选择和生成。与现有数据集不同的是，它包含了清晰的支持知识，以话语为单位分成多个知识来源。我们进一步提出了一个新挑战，即对话知识即插即用，旨在以零-shot的方式测试已经训练好的对话模型在使用未见过的新知识来源时的能力。

一个面向开放领域对话生成的知识即插即用测试平台