The grading of open-ended questions is a high-effort, high-impact task in
education. Automating this task promises a significant reduction in workload
for education professionals, as well as more consistent grading outcomes for
students, by circumventing human subjectivity and error. While recent
breakthroughs in AI technology might facilitate such automation, this has not
been demonstrated at scale. It this paper, we introduce a novel automatic short
answer grading (ASAG) system. The system is based on a fine-tuned open-source
transformer model which we trained on large set of exam data from university
courses across a large range of disciplines. We evaluated the trained model's
performance against held-out test data in a first experiment and found high
accuracy levels across a broad spectrum of unseen questions, even in unseen
courses. We further compared the performance of our model with that of
certified human domain experts in a second experiment: we first assembled
another test dataset from real historical exams - the historic grades contained
in that data were awarded to students in a regulated, legally binding
examination process; we therefore considered them as ground truth for our
experiment. We then asked certified human domain experts and our model to grade
the historic student answers again without disclosing the historic grades.
Finally, we compared the hence obtained grades with the historic grades (our
ground truth). We found that for the courses examined, the model deviated less
from the official historic grades than the human re-graders - the model's
median absolute error was 44 % smaller than the human re-graders', implying
that the model is more consistent than humans in grading. These results suggest
that leveraging AI enhanced grading can reduce human subjectivity, improve
consistency and thus ultimately increase fairness.

通过基于大规模的大学课程考试数据训练的神经网络模型，在历史考试数据上的实验证明了自动化简短回答评分系统的高准确性和一致性，这为减少人为主观性、改善评分一致性以提高公正性提供了有前途的解决方案。