Automatic evaluation of open-domain dialogue response generation is very
challenging because there are many appropriate responses for a given context.
Existing evaluation models merely compare the generated response with the
ground truth response and rate many of the appropriate responses as
inappropriate if they deviate from the ground truth. One approach to resolve
this problem is to consider the similarity of the generated response with the
conversational context. In this paper, we propose an automatic evaluation model
based on that idea and learn the model parameters from an unlabeled
conversation corpus. Our approach considers the speakers in defining the
different levels of similar context. We use a Twitter conversation corpus that
contains many speakers and conversations to test our evaluation model.
Experiments show that our model outperforms the other existing evaluation
metrics in terms of high correlation with human annotation scores. We also show
that our model trained on Twitter can be applied to movie dialogues without any
additional training. We provide our code and the learned parameters so that
they can be used for automatic evaluation of dialogue response generation
models.

该研究提出了一种基于相邻对话的自动评估模型，利用未标注对话语料库中的说话者定义不同级别的类似上下文，使用推特对话语料库进行实验，表明与其他现有评估指标相比，该模型具有更高的人类标注得分相关性。并展示了在无需额外训练的情况下，将该模型应用于电影对话。