Massive open online courses (MOOCs) unlock the doors to free education for
anyone around the globe with access to a computer and the internet. Despite
this democratization of learning, the massive enrollment in these courses means
it is almost impossible for one instructor to assess every student's writing
assignment. As a result, peer grading, often guided by a straightforward
rubric, is the method of choice. While convenient, peer grading often falls
short in terms of reliability and validity. In this study, using 18 distinct
settings, we explore the feasibility of leveraging large language models (LLMs)
to replace peer grading in MOOCs. Specifically, we focus on two
state-of-the-art LLMs: GPT-4 and GPT-3.5, across three distinct courses:
Introductory Astronomy, Astrobiology, and the History and Philosophy of
Astronomy. To instruct LLMs, we use three different prompts based on a variant
of the zero-shot chain-of-thought (Zero-shot-CoT) prompting technique:
Zero-shot-CoT combined with instructor-provided correct answers; Zero-shot-CoT
in conjunction with both instructor-formulated answers and rubrics; and
Zero-shot-CoT with instructor-offered correct answers and LLM-generated
rubrics. Our results show that Zero-shot-CoT, when integrated with
instructor-provided answers and rubrics, produces grades that are more aligned
with those assigned by instructors compared to peer grading. However, the
History and Philosophy of Astronomy course proves to be more challenging in
terms of grading as opposed to other courses. Finally, our study reveals a
promising direction for automating grading systems for MOOCs, especially in
subjects with well-defined rubrics.

使用大型语言模型（LLMs）取代 MOOCs 中的同行评分，通过 Zero-shot-CoT 和多个方案进行教学，发现 Zero-shot-CoT 与教师提供的答案和评分标准相结合的结果与教师评分更加一致，为 MOOCs 自动化评分系统提供了有希望的方向。

大型语言模型作为 MOOC 课程的评分器

Large Language Models As MOOCs Graders

Large Language Models (LLMs) have limited performance when solving arithmetic
reasoning tasks and often provide incorrect answers. Unlike natural language
understanding, math problems typically have a single correct answer, making the
task of generating accurate solutions more challenging for LLMs. To the best of
our knowledge, we are not aware of any LLMs that indicate their level of
confidence in their responses which fuels a trust deficit in these models
impeding their adoption. To address this deficiency, we propose `MathPrompter',
a technique that improves performance of LLMs on arithmetic problems along with
increased reliance in the predictions. MathPrompter uses the Zero-shot
chain-of-thought prompting technique to generate multiple Algebraic expressions
or Python functions to solve the same math problem in different ways and
thereby raise the confidence level in the output results. This is in contrast
to other prompt based CoT methods, where there is no check on the validity of
the intermediate steps followed. Our technique improves over state-of-the-art
on the MultiArith dataset ($78.7\%\rightarrow92.5\%$) evaluated using 175B
parameter GPT-based LLM.

提出了一种名为 MathPrompter 的技术，它使用 Zero-shot chain-of-thought 提示技术生成多个代数表达式或 Python 函数，并以不同的方式解决相同的数学问题，从而提高模型在算术问题上的性能并提高置信水平。