An important unexplored aspect in previous work on user satisfaction
estimation for Task-Oriented Dialogue (TOD) systems is their evaluation in
terms of robustness for the identification of user dissatisfaction: current
benchmarks for user satisfaction estimation in TOD systems are highly skewed
towards dialogues for which the user is satisfied. The effect of having a more
balanced set of satisfaction labels on performance is unknown. However,
balancing the data with more dissatisfactory dialogue samples requires further
data collection and human annotation, which is costly and time-consuming. In
this work, we leverage large language models (LLMs) and unlock their ability to
generate satisfaction-aware counterfactual dialogues to augment the set of
original dialogues of a test collection. We gather human annotations to ensure
the reliability of the generated samples. We evaluate two open-source LLMs as
user satisfaction estimators on our augmented collection against
state-of-the-art fine-tuned models. Our experiments show that when used as
few-shot user satisfaction estimators, open-source LLMs show higher robustness
to the increase in the number of dissatisfaction labels in the test collection
than the fine-tuned state-of-the-art models. Our results shed light on the need
for data augmentation approaches for user satisfaction estimation in TOD
systems. We release our aligned counterfactual dialogues, which are curated by
human annotation, to facilitate further research on this topic.

利用大型语言模型 (LLMs) 生成注重满意度的反事实对话以增加测试集中的原始对话样本，并通过人工注释验证，研究表明，与最先进的微调模型相比，开源的大型语言模型作为少样本的用户满意度评估器，在测试集中不满意标签数量的增加时表现出更高的鲁棒性。