This paper presents a novel application of large language models in user
simulation for task-oriented dialog systems, specifically focusing on an
in-context learning approach. By harnessing the power of these models, the
proposed approach generates diverse utterances based on user goals and limited
dialog examples. Unlike traditional simulators, this method eliminates the need
for labor-intensive rule definition or extensive annotated data, making it more
efficient and accessible. Additionally, an error analysis of the interaction
between the user simulator and dialog system uncovers common mistakes,
providing valuable insights into areas that require improvement. Our
implementation is available at
this https URL

本研究提出了一种基于大型语言模型的用户模拟方法来优化面向任务的对话系统，这种方法通过在用户目标和有限对话示例的基础上产生多样化的话语来消除手工规则定义或广泛注释数据的需要，并对用户模拟器和对话系统之间的相互作用进行了误差分析，提供了改进的有价值见解。

针对任务导向对话系统的上下文学习用户模拟器

In-Context Learning User Simulators for Task-Oriented Dialog Systems

Automatic short answer grading is an important research direction in the
exploration of how to use artificial intelligence (AI)-based tools to improve
education. Current state-of-the-art approaches use neural language models to
create vectorized representations of students responses, followed by
classifiers to predict the score. However, these approaches have several key
limitations, including i) they use pre-trained language models that are not
well-adapted to educational subject domains and/or student-generated text and
ii) they almost always train one model per question, ignoring the linkage
across a question and result in a significant model storage problem due to the
size of advanced language models. In this paper, we study the problem of
automatic short answer grading for students' responses to math questions and
propose a novel framework for this task. First, we use MathBERT, a variant of
the popular language model BERT adapted to mathematical content, as our base
model and fine-tune it for the downstream task of student response grading.
Second, we use an in-context learning approach that provides scoring examples
as input to the language model to provide additional context information and
promote generalization to previously unseen questions. We evaluate our
framework on a real-world dataset of student responses to open-ended math
questions and show that our framework (often significantly) outperforms
existing approaches, especially for new questions that are not seen during
training.

本文研究了自动短答案评分问题，提出了一种新的基于 MathBERT 及上下文学习方法的框架，并在真实数据集上进行了评估，证明该框架对于之前未见过的数学问题的表现优于现有方法。