The conversational recommendation system (CRS) has been criticized regarding
its user experience in real-world scenarios, despite recent significant
progress achieved in academia. Existing evaluation protocols for CRS may
prioritize system-centric factors such as effectiveness and fluency in
conversation while neglecting user-centric aspects. Thus, we propose a new and
inclusive evaluation protocol, Concept, which integrates both system- and
user-centric factors. We conceptualise three key characteristics in
representing such factors and further divide them into six primary abilities.
To implement Concept, we adopt a LLM-based user simulator and evaluator with
scoring rubrics that are tailored for each primary ability. Our protocol,
Concept, serves a dual purpose. First, it provides an overview of the pros and
cons in current CRS models. Second, it pinpoints the problem of low usability
in the "omnipotent" ChatGPT and offers a comprehensive reference guide for
evaluating CRS, thereby setting the foundation for CRS improvement.

当前的对话推荐系统 (CRS) 在现实场景中的用户体验遭受批评，尽管在学术界取得了显著进展。存在着针对 CRS 的评估协议可能偏重系统中心因素，如对话的效果和流畅性，而忽视了用户中心的方面。因此，我们提出了一个新的全面的评估协议 Concept，将系统中心和用户中心因素进行整合。通过概括三个关键特性以及将它们划分为六个主要能力，来实施 Concept。为了实现 Concept，我们采用了基于 LLM 的用户模拟器和评估器，并为每个主要能力定制了评分标准。我们的协议 Concept 有两个目的。首先，它提供了当前 CRS 模型中优点和缺点的概述。其次，它指出了 “无所不能” 的 ChatGPT 中可用性问题，并为评估 CRS 提供了全面的参考指南，从而为 CRS 的改进奠定了基础。

基于系统和用户角度的对话推荐系统评估协议

Concept -- An Evaluation Protocol on Conversation Recommender Systems  with System- and User-centric Factors

A Conversational Recommender System (CRS) offers increased transparency and
control to users by enabling them to engage with the system through a real-time
multi-turn dialogue. Recently, Large Language Models (LLMs) have exhibited an
unprecedented ability to converse naturally and incorporate world knowledge and
common-sense reasoning into language understanding, unlocking the potential of
this paradigm. However, effectively leveraging LLMs within a CRS introduces new
technical challenges, including properly understanding and controlling a
complex conversation and retrieving from external sources of information. These
issues are exacerbated by a large, evolving item corpus and a lack of
conversational data for training. In this paper, we provide a roadmap for
building an end-to-end large-scale CRS using LLMs. In particular, we propose
new implementations for user preference understanding, flexible dialogue
management and explainable recommendations as part of an integrated
architecture powered by LLMs. For improved personalization, we describe how an
LLM can consume interpretable natural language user profiles and use them to
modulate session-level context. To overcome conversational data limitations in
the absence of an existing production CRS, we propose techniques for building a
controllable LLM-based user simulator to generate synthetic conversations. As a
proof of concept we introduce RecLLM, a large-scale CRS for YouTube videos
built on LaMDA, and demonstrate its fluency and diverse functionality through
some illustrative example conversations.

本文提出了一个使用大型语言模型的端到端大规模语音推荐系统，并介绍了一些技术和实现细节，包括用户偏好理解、灵活对话管理、基于 LLM 的用户仿真器等。