The conversational recommendation system (CRS) has been criticized regarding
its user experience in real-world scenarios, despite recent significant
progress achieved in academia. Existing evaluation protocols for CRS may
prioritize system-centric factors such as effectiveness and fluency in
conversation while neglecting user-centric aspects. Thus, we propose a new and
inclusive evaluation protocol, Concept, which integrates both system- and
user-centric factors. We conceptualise three key characteristics in
representing such factors and further divide them into six primary abilities.
To implement Concept, we adopt a LLM-based user simulator and evaluator with
scoring rubrics that are tailored for each primary ability. Our protocol,
Concept, serves a dual purpose. First, it provides an overview of the pros and
cons in current CRS models. Second, it pinpoints the problem of low usability
in the "omnipotent" ChatGPT and offers a comprehensive reference guide for
evaluating CRS, thereby setting the foundation for CRS improvement.

当前的对话推荐系统 (CRS) 在现实场景中的用户体验遭受批评，尽管在学术界取得了显著进展。存在着针对 CRS 的评估协议可能偏重系统中心因素，如对话的效果和流畅性，而忽视了用户中心的方面。因此，我们提出了一个新的全面的评估协议 Concept，将系统中心和用户中心因素进行整合。通过概括三个关键特性以及将它们划分为六个主要能力，来实施 Concept。为了实现 Concept，我们采用了基于 LLM 的用户模拟器和评估器，并为每个主要能力定制了评分标准。我们的协议 Concept 有两个目的。首先，它提供了当前 CRS 模型中优点和缺点的概述。其次，它指出了 “无所不能” 的 ChatGPT 中可用性问题，并为评估 CRS 提供了全面的参考指南，从而为 CRS 的改进奠定了基础。