Accurate automatic evaluation metrics for open-domain dialogs are in high
demand. Existing model-based metrics for system response evaluation are trained
on human annotated data, which is cumbersome to collect. In this work, we
propose to use information that can be automatically extracted from the next
user utterance, such as its sentiment or whether the user explicitly ends the
conversation, as a proxy to measure the quality of the previous system
response. This allows us to train on a massive set of dialogs with weak
supervision, without requiring manual system turn quality annotations.
Experiments show that our model is comparable to models trained on human
annotated data. Furthermore, our model generalizes across both spoken and
written open-domain dialog corpora collected from real and paid users.

本文提出使用自动提取的信息作为衡量已有机器人回答质量的替代方法，从而避免了对人工标注数据的依赖，实现对弱监督数据的训练，并添加了对口语和书面语的支持。