Massive open online courses (MOOCs) unlock the doors to free education for
anyone around the globe with access to a computer and the internet. Despite
this democratization of learning, the massive enrollment in these courses means
it is almost impossible for one instructor to assess every student's writing
assignment. As a result, peer grading, often guided by a straightforward
rubric, is the method of choice. While convenient, peer grading often falls
short in terms of reliability and validity. In this study, using 18 distinct
settings, we explore the feasibility of leveraging large language models (LLMs)
to replace peer grading in MOOCs. Specifically, we focus on two
state-of-the-art LLMs: GPT-4 and GPT-3.5, across three distinct courses:
Introductory Astronomy, Astrobiology, and the History and Philosophy of
Astronomy. To instruct LLMs, we use three different prompts based on a variant
of the zero-shot chain-of-thought (Zero-shot-CoT) prompting technique:
Zero-shot-CoT combined with instructor-provided correct answers; Zero-shot-CoT
in conjunction with both instructor-formulated answers and rubrics; and
Zero-shot-CoT with instructor-offered correct answers and LLM-generated
rubrics. Our results show that Zero-shot-CoT, when integrated with
instructor-provided answers and rubrics, produces grades that are more aligned
with those assigned by instructors compared to peer grading. However, the
History and Philosophy of Astronomy course proves to be more challenging in
terms of grading as opposed to other courses. Finally, our study reveals a
promising direction for automating grading systems for MOOCs, especially in
subjects with well-defined rubrics.

使用大型语言模型（LLMs）取代 MOOCs 中的同行评分，通过 Zero-shot-CoT 和多个方案进行教学，发现 Zero-shot-CoT 与教师提供的答案和评分标准相结合的结果与教师评分更加一致，为 MOOCs 自动化评分系统提供了有希望的方向。