Dialog evaluation is a challenging problem, especially for non task-oriented
dialogs where conversational success is not well-defined. We propose to
evaluate dialog quality using topic-based metrics that describe the ability of
a conversational bot to sustain coherent and engaging conversations on a topic,
and the diversity of topics that a bot can handle. To detect conversation
topics per utterance, we adopt Deep Average Networks (DAN) and train a topic
classifier on a variety of question and query data categorized into multiple
topics. We propose a novel extension to DAN by adding a topic-word attention
table that allows the system to jointly capture topic keywords in an utterance
and perform topic classification. We compare our proposed topic based metrics
with the ratings provided by users and show that our metrics both correlate
with and complement human judgment. Our analysis is performed on tens of
thousands of real human-bot dialogs from the Alexa Prize competition and
highlights user expectations for conversational bots.

使用基于话题的度量标准来评估对话质量，包括考虑对话机器人在话题上维持连贯和有吸引力的对话能力及其多样性，并采用深度平均网络和话题分类器检测每个话语的对话话题，进一步引入话题关注表以捕捉话语中的话题关键字以及执行话题分类。经过与用户提供的评分进行比较，研究表明这些度量标准既与人类判断相关又补充人类判断，并且在亚历山大奖竞赛中对数万个真实人 - 机器人对话进行分析，凸显出用户对话机器人的期望。