Intelligent Tutoring Systems (ITSs) often contain an automated feedback
component, which provides a predefined feedback message to students when they
detect a predefined error. To such a feedback component, we often resort to
template-based approaches. These approaches require significant effort from
human experts to detect a limited number of possible student errors and provide
corresponding feedback. This limitation is exemplified in open-ended math
questions, where there can be a large number of different incorrect errors. In
our work, we examine the capabilities of large language models (LLMs) to
generate feedback for open-ended math questions, similar to that of an
established ITS that uses a template-based approach. We fine-tune both
open-source and proprietary LLMs on real student responses and corresponding
ITS-provided feedback. We measure the quality of the generated feedback using
text similarity metrics. We find that open-source and proprietary models both
show promise in replicating the feedback they see during training, but do not
generalize well to previously unseen student errors. These results suggest that
despite being able to learn the formatting of feedback, LLMs are not able to
fully understand mathematical errors made by students.

智能辅导系统中的自动反馈组件通常采用基于模板的方法，但在开放式数学问题中存在大量不同的错误类型。本研究使用大型语言模型来生成开放式数学问题的反馈，发现尽管能够学习反馈的格式，但无法完全理解学生的数学错误。

大型语言模型能否复制开放式数学问题上的 ITS 反馈？

Can Large Language Models Replicate ITS Feedback on Open-Ended Math  Questions?

Automatic short answer grading is an important research direction in the
exploration of how to use artificial intelligence (AI)-based tools to improve
education. Current state-of-the-art approaches use neural language models to
create vectorized representations of students responses, followed by
classifiers to predict the score. However, these approaches have several key
limitations, including i) they use pre-trained language models that are not
well-adapted to educational subject domains and/or student-generated text and
ii) they almost always train one model per question, ignoring the linkage
across a question and result in a significant model storage problem due to the
size of advanced language models. In this paper, we study the problem of
automatic short answer grading for students' responses to math questions and
propose a novel framework for this task. First, we use MathBERT, a variant of
the popular language model BERT adapted to mathematical content, as our base
model and fine-tune it for the downstream task of student response grading.
Second, we use an in-context learning approach that provides scoring examples
as input to the language model to provide additional context information and
promote generalization to previously unseen questions. We evaluate our
framework on a real-world dataset of student responses to open-ended math
questions and show that our framework (often significantly) outperforms
existing approaches, especially for new questions that are not seen during
training.

本文研究了自动短答案评分问题，提出了一种新的基于 MathBERT 及上下文学习方法的框架，并在真实数据集上进行了评估，证明该框架对于之前未见过的数学问题的表现优于现有方法。