User ratings play a significant role in spoken dialogue systems. Typically,
such ratings tend to be averaged across all users and then utilized as feedback
to improve the system or personalize its behavior. While this method can be
useful to understand broad, general issues with the system and its behavior, it
does not take into account differences between users that affect their ratings.
In this work, we conduct a study to better understand how people rate their
interactions with conversational agents. One macro-level characteristic that
has been shown to correlate with how people perceive their inter-personal
communication is personality. We specifically focus on agreeableness and
extraversion as variables that may explain variation in ratings and therefore
provide a more meaningful signal for training or personalization. In order to
elicit those personality traits during an interaction with a conversational
agent, we designed and validated a fictional story, grounded in prior work in
psychology. We then implemented the story into an experimental conversational
agent that allowed users to opt-in to hearing the story. Our results suggest
that for human-conversational agent interactions, extraversion may play a role
in user ratings, but more data is needed to determine if the relationship is
significant. Agreeableness, on the other hand, plays a statistically
significant role in conversation ratings: users who are more agreeable are more
likely to provide a higher rating for their interaction. In addition, we found
that users who opted to hear the story were, in general, more likely to rate
their conversational experience higher than those who did not.

该研究旨在探讨用户评分在口语对话系统中的作用，特别关注用户的人格特质如宜人性和外向性对于交互评分的影响，并设计和验证了一个虚构故事，以此来调查和诱发这些人格特质。结果表明宜人性与交互评分有着显著的相关性，而选择听故事的用户相对来说更可能提供更高的评分。