Accurate and interpretable user satisfaction estimation (USE) is critical for
understanding, evaluating, and continuously improving conversational systems.
Users express their satisfaction or dissatisfaction with diverse conversational
patterns in both general-purpose (ChatGPT and Bing Copilot) and task-oriented
(customer service chatbot) conversational systems. Existing approaches based on
featurized ML models or text embeddings fall short in extracting generalizable
patterns and are hard to interpret. In this work, we show that LLMs can extract
interpretable signals of user satisfaction from their natural language
utterances more effectively than embedding-based approaches. Moreover, an LLM
can be tailored for USE via an iterative prompting framework using supervision
from labeled examples. The resulting method, Supervised Prompting for User
satisfaction Rubrics (SPUR), not only has higher accuracy but is more
interpretable as it scores user satisfaction via learned rubrics with a
detailed breakdown.

使用迭代提示框架和标记示例的监督下，通过学习得到的评分标准，文中提出了一种更高准确率且更易解释的用户满意度估计方法 ——SPUR，并证明了 LLMs 在从自然语言表达中更有效地提取用户满意度信号。

大规模语言模型中可解释的对话系统用户满意度评估

Interpretable User Satisfaction Estimation for Conversational Systems  with Large Language Models

Group-based sparsity models are proven instrumental in linear regression
problems for recovering signals from much fewer measurements than standard
compressive sensing. The main promise of these models is the recovery of
"interpretable" signals through the identification of their constituent groups.
In this paper, we establish a combinatorial framework for group-model selection
problems and highlight the underlying tractability issues. In particular, we
show that the group-model selection problem is equivalent to the well-known
NP-hard weighted maximum coverage problem (WMC). Leveraging a graph-based
understanding of group models, we describe group structures which enable
correct model selection in polynomial time via dynamic programming.
Furthermore, group structures that lead to totally unimodular constraints have
tractable discrete as well as convex relaxations. We also present a
generalization of the group-model that allows for within group sparsity, which
can be used to model hierarchical sparsity. Finally, we study the Pareto
frontier of group-sparse approximations for two tractable models, among which
the tree sparsity model, and illustrate selection and computation trade-offs
between our framework and the existing convex relaxations.

本研究通过图形模型的理解和动态规划技术，基于组稀疏模型的组结构，解决了组模型选择问题，提出了一种修正的组模型，并研究了两个可操作的模型的群稀疏逼近的 Pareto 前沿以及选择和计算的权衡。