Automated Short Answer Grading (ASAG) has been an active area of
machine-learning research for over a decade. It promises to let educators grade
and give feedback on free-form responses in large-enrollment courses in spite
of limited availability of human graders. Over the years, carefully trained
models have achieved increasingly higher levels of performance. More recently,
pre-trained Large Language Models (LLMs) emerged as a commodity, and an
intriguing question is how a general-purpose tool without additional training
compares to specialized models. We studied the performance of GPT-4 on the
standard benchmark 2-way and 3-way datasets SciEntsBank and Beetle, where in
addition to the standard task of grading the alignment of the student answer
with a reference answer, we also investigated withholding the reference answer.
We found that overall, the performance of the pre-trained general-purpose GPT-4
LLM is comparable to hand-engineered models, but worse than pre-trained LLMs
that had specialized training.

自动短答案评分（ASAG）是一个活跃的机器学习研究领域已有十多年的时间。它承诺即使在人工评分师有限的情况下，让教育者对大班课中的自由回答进行评分和反馈。近年来，经过精心训练的模型已经取得了越来越高的性能水平。最近，预训练的大型语言模型（LLMs）作为一种通用工具出现了，并且引发了一个有趣的问题，即没有额外训练的通用工具与专门模型相比如何。我们研究了 GPT-4 在标准基准 2 路和 3 路数据集 SciEntsBank 和 Beetle 上的性能，除了评分学生答案与参考答案的对齐标准任务外，还研究了不透露参考答案的情况。我们发现，总体而言，预训练的通用 GPT-4 LLM 的性能与手工设计的模型相当，但比经过专门训练的 LLMs 差。

GPT-4 大型预训练语言模型在自动化短答案评分中的表现

Performance of the Pre-Trained Large Language Model GPT-4 on Automated  Short Answer Grading

In this study, we developed an automated short answer grading (ASAG) model
that provided both analytic scores and final holistic scores. Short answer
items typically consist of multiple sub-questions, and providing an analytic
score and the text span relevant to each sub-question can increase the
interpretability of the automated scores. Furthermore, they can be used to
generate actionable feedback for students. Despite these advantages, most
studies have focused on predicting only holistic scores due to the difficulty
in constructing dataset with manual annotations. To address this difficulty, we
used large language model (LLM)-based one-shot prompting and a text similarity
scoring model with domain adaptation using small manually annotated dataset.
The accuracy and quadratic weighted kappa of our model were 0.67 and 0.71 on a
subset of the publicly available ASAG dataset. The model achieved a substantial
improvement over the majority baseline.

本文研究了一种自动短答案评分（ASAG）模型，提供了分析性得分和最终的整体得分。使用基于大型语言模型（LLM）的一次提示和文本相似性评分模型，通过小型手动注释数据集进行领域适应性，获得了较高的准确性和加权 kappa 值，从而在公开 ASAG 数据集的子集上取得了重大进展。

利用单次提示和文本相似性评分模型进行简答评分

Short Answer Grading Using One-shot Prompting and Text Similarity  Scoring Model

Automated short answer grading (ASAG) has gained attention in education as a
means to scale educational tasks to the growing number of students. Recent
progress in Natural Language Processing and Machine Learning has largely
influenced the field of ASAG, of which we survey the recent research
advancements. We complement previous surveys by providing a comprehensive
analysis of recently published methods that deploy deep learning approaches. In
particular, we focus our analysis on the transition from hand engineered
features to representation learning approaches, which learn representative
features for the task at hand automatically from large corpora of data. We
structure our analysis of deep learning methods along three categories: word
embeddings, sequential models, and attention-based methods. Deep learning
impacted ASAG differently than other fields of NLP, as we noticed that the
learned representations alone do not contribute to achieve the best results,
but they rather show to work in a complementary way with hand-engineered
features. The best performance are indeed achieved by methods that combine the
carefully hand-engineered features with the power of the semantic descriptions
provided by the latest models, like transformers architectures. We identify
challenges and provide an outlook on research direction that can be addressed
in the future

本研究综述了最近自然语言处理和机器学习方面的进展，特别是着重于自动化简短答案评分的深度学习方法。最佳表现是通过结合精心设计的手工特征和变压器结构提供的语义描述来实现的。