Conversational information access is an emerging research area. Currently,
human evaluation is used for end-to-end system evaluation, which is both very
time and resource intensive at scale, and thus becomes a bottleneck of
progress. As an alternative, we propose automated evaluation by means of
simulating users. Our user simulator aims to generate responses that a real
human would give by considering both individual preferences and the general
flow of interaction with the system. We evaluate our simulation approach on an
item recommendation task by comparing three existing conversational recommender
systems. We show that preference modeling and task-specific interaction models
both contribute to more realistic simulations, and can help achieve high
correlation between automatic evaluation measures and manual human assessments.

本文提出一种通过用户模拟来替代人类评估的方法，以实现对话推荐系统的自动评估。作者表明，通过考虑用户的个人喜好和与系统的一般交互流程，偏好建模和特定任务交互模型都有助于实现高度自动化的绩效评估结果和人工人工评估的高度相关性。