The aim of this paper is to mitigate the shortcomings of automatic evaluation
of open-domain dialog systems through multi-reference evaluation. Existing
metrics have been shown to correlate poorly with human judgement, particularly
in open-domain dialog. One alternative is to collect human annotations for
evaluation, which can be expensive and time consuming. To demonstrate the
effectiveness of multi-reference evaluation, we augment the test set of
DailyDialog with multiple references. A series of experiments show that the use
of multiple references results in improved correlation between several
automatic metrics and human judgement for both the quality and the diversity of
system output.

本文旨在通过多参考评估减少开放领域对话系统的自动评估的缺点，研究结果表明，使用多参考评估可以提高几种自动指标与人类判断之间的相关性，无论是对于系统输出的质量还是多样性。

通过人工多重参考研究开放式对话系统的评估

Investigating Evaluation of Open-Domain Dialogue Systems With Human  Generated Multiple References

We develop a high-quality multi-turn dialog dataset, DailyDialog, which is
intriguing in several aspects. The language is human-written and less noisy.
The dialogues in the dataset reflect our daily communication way and cover
various topics about our daily life. We also manually label the developed
dataset with communication intention and emotion information. Then, we evaluate
existing approaches on DailyDialog dataset and hope it benefit the research
field of dialog systems.

我们开发了一个高质量的多回合对话数据集 DailyDialog，包括人类编写的不太嘈杂的语言，反映我们日常交流方式并涵盖我们日常生活的各种主题。我们手动标记了这个数据集的通信意图和情感信息，并在 DailyDialog 数据集上评估现有的方法，希望它有益于对话系统的研究领域。