Grading of exams is an important, labor intensive, subjective, repetitive and
frequently challenging task. The feasibility of autograding textual responses
has greatly increased thanks to the availability of large language models
(LLMs) such as ChatGPT and because of the substantial influx of data brought
about by digitalization. However, entrusting AI models with decision-making
roles raises ethical considerations, mainly stemming from potential biases and
issues related to generating false information. Thus, in this manuscript we
provide an evaluation of a large language model for the purpose of autograding,
while also highlighting how LLMs can support educators in validating their
grading procedures. Our evaluation is targeted towards automatic short textual
answers grading (ASAG), spanning various languages and examinations from two
distinct courses. Our findings suggest that while "out-of-the-box" LLMs provide
a valuable tool to provide a complementary perspective, their readiness for
independent automated grading remains a work in progress, necessitating human
oversight.

通过评估大型语言模型在自动评分方面的可行性，并强调大型语言模型如何支持教育工作者验证评分程序，研究表明，虽然 “开箱即用” 的大型语言模型提供了宝贵的工具来提供补充视角，但它们对于独立自动评分的准备工作仍然是一个尚未完成的工作，需要人工监督。

基于 LLM 的短文本答案自动评分方法探究

Towards LLM-based Autograding for Short Textual Answers

Automated short answer grading (ASAG) has gained attention in education as a
means to scale educational tasks to the growing number of students. Recent
progress in Natural Language Processing and Machine Learning has largely
influenced the field of ASAG, of which we survey the recent research
advancements. We complement previous surveys by providing a comprehensive
analysis of recently published methods that deploy deep learning approaches. In
particular, we focus our analysis on the transition from hand engineered
features to representation learning approaches, which learn representative
features for the task at hand automatically from large corpora of data. We
structure our analysis of deep learning methods along three categories: word
embeddings, sequential models, and attention-based methods. Deep learning
impacted ASAG differently than other fields of NLP, as we noticed that the
learned representations alone do not contribute to achieve the best results,
but they rather show to work in a complementary way with hand-engineered
features. The best performance are indeed achieved by methods that combine the
carefully hand-engineered features with the power of the semantic descriptions
provided by the latest models, like transformers architectures. We identify
challenges and provide an outlook on research direction that can be addressed
in the future

本研究综述了最近自然语言处理和机器学习方面的进展，特别是着重于自动化简短答案评分的深度学习方法。最佳表现是通过结合精心设计的手工特征和变压器结构提供的语义描述来实现的。