Existing automatic evaluation metrics for open-domain dialogue response
generation systems correlate poorly with human evaluation. We focus on
evaluating response generation systems via response selection. To evaluate
systems properly via response selection, we propose the method to construct
response selection test sets with well-chosen false candidates. Specifically,
we propose to construct test sets filtering out some types of false candidates:
(i) those unrelated to the ground-truth response and (ii) those acceptable as
appropriate responses. Through experiments, we demonstrate that evaluating
systems via response selection with the test sets developed by our method
correlates more strongly with human evaluation, compared with widely used
automatic evaluation metrics such as BLEU.

本研究提出了一种构建响应选择测试集以进行系统评估的方法，通过过滤不相关的和可接受的候选者，实验表明此方法比 BLEU 等自动评估指标更能与人类评估相关联。

通过响应选择评估对话生成系统

Evaluating Dialogue Generation Systems via Response Selection

Although deep learning models have brought tremendous advancements to the
field of open-domain dialogue response generation, recent research results have
revealed that the trained models have undesirable generation behaviors, such as
malicious responses and generic (boring) responses. In this work, we propose a
framework named "Negative Training" to minimize such behaviors. Given a trained
model, the framework will first find generated samples that exhibit the
undesirable behavior, and then use them to feed negative training signals for
fine-tuning the model. Our experiments show that negative training can
significantly reduce the hit rate of malicious responses, or discourage
frequent responses and improve response diversity.

本文提出了一种名为 “负面训练” 的框架，通过寻找令人不满意的行为来微调模型，以显著减少恶意回应，或抑制频繁回应并提高回应多样性。