Knowledge-based, open-domain dialogue generation aims to build chit-chat
systems that talk to humans using mined support knowledge. Many types and
sources of knowledge have previously been shown to be useful as support
knowledge. Even in the era of large language models, response generation
grounded in knowledge retrieved from additional up-to-date sources remains a
practically important approach. While prior work using single-source knowledge
has shown a clear positive correlation between the performances of knowledge
selection and response generation, there are no existing multi-source datasets
for evaluating support knowledge retrieval. Further, prior work has assumed
that the knowledge sources available at test time are the same as during
training. This unrealistic assumption unnecessarily handicaps models, as new
knowledge sources can become available after a model is trained. In this paper,
we present a high-quality benchmark named multi-source Wizard of Wikipedia
(Ms.WoW) for evaluating multi-source dialogue knowledge selection and response
generation. Unlike existing datasets, it contains clean support knowledge,
grounded at the utterance level and partitioned into multiple knowledge
sources. We further propose a new challenge, dialogue knowledge plug-and-play,
which aims to test an already trained dialogue model on using new support
knowledge from previously unseen sources in a zero-shot fashion.

构建基于知识的开放领域对话生成技术旨在利用挖掘的知识与人类进行闲聊。本文提出了一个高质量的多源对话知识选择和生成评估基准，名为 multi-source Wizard of Wikipedia（Ms.WoW），用于评估多源对话知识选择和生成。与现有数据集不同的是，它包含了清晰的支持知识，以话语为单位分成多个知识来源。我们进一步提出了一个新挑战，即对话知识即插即用，旨在以零 - shot 的方式测试已经训练好的对话模型在使用未见过的新知识来源时的能力。