Establishing evaluation schemes for spoken dialogue systems is important, but
it can also be challenging. While subjective evaluations are commonly used in
user experiments, objective evaluations are necessary for research comparison
and reproducibility. To address this issue, we propose a framework for
indirectly but objectively evaluating systems based on users' behaviours. In
this paper, to this end, we investigate the relationship between user
behaviours and subjective evaluation scores in social dialogue tasks: attentive
listening, job interview, and first-meeting conversation. The results reveal
that in dialogue tasks where user utterances are primary, such as attentive
listening and job interview, indicators like the number of utterances and words
play a significant role in evaluation. Observing disfluency also can indicate
the effectiveness of formal tasks, such as job interview. On the other hand, in
dialogue tasks with high interactivity, such as first-meeting conversation,
behaviours related to turn-taking, like average switch pause length, become
more important. These findings suggest that selecting appropriate user
behaviours can provide valuable insights for objective evaluation in each
social dialogue task.

间接但客观地评估系统基于用户行为的框架在社交对话任务中揭示了用户行为与主观评估分数之间的关系，并提出了选择适当的用户行为以为每个社交对话任务进行客观评估提供有价值的观察。

分析用户行为以客观评估口语对话系统

An Analysis of User Behaviours for Objectively Evaluating Spoken  Dialogue Systems

Human ratings are one of the most prevalent methods to evaluate the
performance of natural language processing algorithms. Similarly, it is common
to measure the quality of sentences generated by a natural language generation
model using human raters. In this paper, we argue for exploring the use of
subjective evaluations within the process of training language generation
models in a multi-task learning setting. As a case study, we use a
crowd-authored dialogue corpus to fine-tune six different language generation
models. Two of these models incorporate multi-task learning and use subjective
ratings of lines as part of an explicit learning goal. A human evaluation of
the generated dialogue lines reveals that utterances generated by the
multi-tasking models were subjectively rated as the most typical, most moving
the conversation forward, and least offensive. Based on these promising first
results, we discuss future research directions for incorporating subjective
human evaluations into language model training and to hence keep the human user
in the loop during the development process.

本文探讨在多任务学习设置中，使用主观评估作为语言生成模型训练的一部分，并使用群众创作对话语料库对六种不同的语言生成模型进行微调。评估显示，多任务学习的模型生成的话语在主观上评分最高，且在推动对话发展、无冒犯性等方面得分最高。因此，将来可以研究将主观人类评估纳入语言生成模型训练中，从而在开发过程中更好地与人类用户进行交互。