Establishing evaluation schemes for spoken dialogue systems is important, but
it can also be challenging. While subjective evaluations are commonly used in
user experiments, objective evaluations are necessary for research comparison
and reproducibility. To address this issue, we propose a framework for
indirectly but objectively evaluating systems based on users' behaviours. In
this paper, to this end, we investigate the relationship between user
behaviours and subjective evaluation scores in social dialogue tasks: attentive
listening, job interview, and first-meeting conversation. The results reveal
that in dialogue tasks where user utterances are primary, such as attentive
listening and job interview, indicators like the number of utterances and words
play a significant role in evaluation. Observing disfluency also can indicate
the effectiveness of formal tasks, such as job interview. On the other hand, in
dialogue tasks with high interactivity, such as first-meeting conversation,
behaviours related to turn-taking, like average switch pause length, become
more important. These findings suggest that selecting appropriate user
behaviours can provide valuable insights for objective evaluation in each
social dialogue task.

间接但客观地评估系统基于用户行为的框架在社交对话任务中揭示了用户行为与主观评估分数之间的关系，并提出了选择适当的用户行为以为每个社交对话任务进行客观评估提供有价值的观察。

分析用户行为以客观评估口语对话系统

An Analysis of User Behaviours for Objectively Evaluating Spoken  Dialogue Systems

Despite the rapid advancement of mobile applications, predicting app usage
remains a formidable challenge due to intricate user behaviours and
ever-evolving contexts. To address these issues, this paper introduces the
Mobile App Prediction Leveraging Large Language Model Embeddings (MAPLE) model.
This innovative approach utilizes Large Language Models (LLMs) to predict app
usage accurately. Rigorous testing on two public datasets highlights MAPLE's
capability to decipher intricate patterns and comprehend user contexts. These
robust results confirm MAPLE's versatility and resilience across various
scenarios. While its primary design caters to app prediction, the outcomes also
emphasize the broader applicability of LLMs in different domains. Through this
research, we emphasize the potential of LLMs in app usage prediction and
suggest their transformative capacity in modelling human behaviours across
diverse fields.

本研究提出了基于大型语言模型嵌入的移动应用预测模型（MAPLE），以应对移动应用预测中复杂的用户行为和不断演化的环境。通过对两个公共数据集的严格测试，结果表明 MAPLE 能够准确解读复杂模式并理解用户背景。这些强大的结果确认了 MAPLE 在不同场景中的多功能性和适应性。虽然其主要设计面向应用预测，但研究结果也强调了大型语言模型在不同领域中的广泛适用性。通过这项研究，我们强调了大型语言模型在应用使用预测中的潜力，并建议它们在模拟人类行为方面具有改变性的能力。