Once powerful conversational models have become available for a wide
audience, users started actively engaging in social interactions with this
technology. Such unprecedented interaction experiences may pose considerable
social and psychological risks to the users unless the technology is properly
controlled. This creates an urgent need for scalable and robust evaluation
metrics for conversational chatbots. Existing automatic evaluation metrics
usually focus on objective quality measures and disregard subjective
perceptions of social dimensions. Moreover, most of these approaches operate on
pre-produced dialogs from available benchmark corpora, which implies human
involvement for preparing the material for evaluation and, thus, impeded
scalability of the metrics. To address this limitation, we propose to make use
of the emerging large language models (LLMs) from the GPT-family and describe a
new framework allowing to conduct dialog system evaluation with prompting. With
this framework, we are able to achieve full automation of the evaluation
pipeline and reach impressive correlation with the human judgement (up to
Pearson r=0.95 on system level). The underlying concept is to collect synthetic
chat logs of evaluated bots with a LLM in the other-play setting, where LLM is
carefully conditioned to follow a specific scenario. We further explore
different prompting approaches to produce evaluation scores with the same LLM.
The best-performing prompts, containing few-show demonstrations and
instructions, show outstanding performance on the tested dataset and
demonstrate the ability to generalize to other dialog corpora.

提出了一种利用 GPT 模型进行对话系统评估的新框架，通过对模型进行特定条件训练来生成评估指标，采用少量的演示和指导进行提示，可以在自动化评估过程中取得与人类评价高度相关的结果。

通过提示逼近人类对社交聊天机器人的评估

Approximating Human Evaluation of Social Chatbots with Prompting

Carefully-designed schemas describing how to collect and annotate dialog
corpora are a prerequisite towards building task-oriented dialog systems. In
practical applications, manually designing schemas can be error-prone,
laborious, iterative, and slow, especially when the schema is complicated. To
alleviate this expensive and time consuming process, we propose an unsupervised
approach for slot schema induction from unlabeled dialog corpora. Leveraging
in-domain language models and unsupervised parsing structures, our data-driven
approach extracts candidate slots without constraints, followed by
coarse-to-fine clustering to induce slot types. We compare our method against
several strong supervised baselines, and show significant performance
improvement in slot schema induction on MultiWoz and SGD datasets. We also
demonstrate the effectiveness of induced schemas on downstream applications
including dialog state tracking and response generation.

提出了一种无监督的方法来从未标注的对话语料库中识别 slot schema，通过在特定领域的语言模型和无监督的解析结构的帮助下提取候选槽位，并通过从粗到细的聚类来诱导槽类型，显示了该方法在 MultiWoz 和 SGD 数据集上诱导槽模式的显着性能改进，并证明了诱导槽模式对于对话状态跟踪和响应生成的有效性。