Automatic evaluation of open-domain dialogs remains an unsolved problem.
Moreover, existing methods do not correlate strongly with human annotations.
This paper presents a new automated evaluation method using follow-ups: we
measure the probability that a language model will continue the conversation
with a fixed set of follow-ups (e.g., not really relevant here, what are you
trying to say). When compared against twelve existing methods, our new
evaluation achieves the highest correlation with human evaluations.

本研究提出一种新的自动对话评估方法，使用固定的跟进集合（例如，这里没有真正相关的内容，你想说什么），测量语言模型继续对话的概率。相对于现有的 12 种评估方法，本方法与人类评估的相关性最高。

使用跟进概率进行开放域对话评估

Open-Domain Dialog Evaluation using Follow-Ups Likelihood

Accurate automatic evaluation metrics for open-domain dialogs are in high
demand. Existing model-based metrics for system response evaluation are trained
on human annotated data, which is cumbersome to collect. In this work, we
propose to use information that can be automatically extracted from the next
user utterance, such as its sentiment or whether the user explicitly ends the
conversation, as a proxy to measure the quality of the previous system
response. This allows us to train on a massive set of dialogs with weak
supervision, without requiring manual system turn quality annotations.
Experiments show that our model is comparable to models trained on human
annotated data. Furthermore, our model generalizes across both spoken and
written open-domain dialog corpora collected from real and paid users.

本文提出使用自动提取的信息作为衡量已有机器人回答质量的替代方法，从而避免了对人工标注数据的依赖，实现对弱监督数据的训练，并添加了对口语和书面语的支持。

利用用户情感进行自动对话评估

What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation

Various encoder-decoder models have been applied to response generation in
open-domain dialogs, but a majority of conventional models directly learn a
mapping from lexical input to lexical output without explicitly modeling
intermediate representations. Utilizing language hierarchy and modeling
intermediate information have been shown to benefit many language understanding
and generation tasks. Motivated by Broca's aphasia, we propose to use a content
word sequence as an intermediate representation for open-domain response
generation. Experimental results show that the proposed method improves content
relatedness of produced responses, and our models can often choose correct
grammar for generated content words. Meanwhile, instead of evaluating complete
sentences, we propose to compute conventional metrics on content word
sequences, which is a better indicator of content relevance.

研究了基于编码器 - 解码器模型的开放领域对话系统的自动回复方法，并提出一种将内容单词序列作为中间表示的方法，并通过将学习和评估的重点从完整句子转移到内容单词序列，提高了生成语句相关性的方法。