While large language models (LLMs) have been used for automated grading, they
have not yet achieved the same level of performance as humans, especially when
it comes to grading complex questions. Existing research on this topic focuses
on a particular step in the grading procedure: grading using predefined
rubrics. However, grading is a multifaceted procedure that encompasses other
crucial steps, such as grading rubrics design and post-grading review. There
has been a lack of systematic research exploring the potential of LLMs to
enhance the entire grading~process.
In this paper, we propose an LLM-based grading system that addresses the
entire grading procedure, including the following key components: 1) Developing
grading rubrics that not only consider the questions but also the student
answers, which can more accurately reflect students' performance. 2) Under the
guidance of grading rubrics, providing accurate and consistent scores for each
student, along with customized feedback. 3) Conducting post-grading review to
better ensure accuracy and fairness. Additionally, we collected a new dataset
named OS from a university operating system course and conducted extensive
experiments on both our new dataset and the widely used Mohler dataset.
Experiments demonstrate the effectiveness of our proposed approach, providing
some new insights for developing automated grading systems based on LLMs.

我们提出了一个基于大型语言模型的评分系统，包括开发评分标准，提供准确一致的得分和定制化反馈，以及进行后评估，并在新的数据集上进行了广泛实验，验证了我们的方法的有效性。