Massive open online courses (MOOCs) offer free education globally to anyone
with a computer and internet access. Despite this democratization of learning,
the massive enrollment in these courses makes it impractical for one instructor
to assess every student's writing assignment. As a result, peer grading, often
guided by a straightforward rubric, is the method of choice. While convenient,
peer grading often falls short in terms of reliability and validity. In this
study, we explore the feasibility of using large language models (LLMs) to
replace peer grading in MOOCs. Specifically, we use two LLMs, GPT-4 and
GPT-3.5, across three MOOCs: Introductory Astronomy, Astrobiology, and the
History and Philosophy of Astronomy. To instruct LLMs, we use three different
prompts based on the zero-shot chain-of-thought (ZCoT) prompting technique: (1)
ZCoT with instructor-provided correct answers, (2) ZCoT with both
instructor-provided correct answers and rubrics, and (3) ZCoT with
instructor-provided correct answers and LLM-generated rubrics. Tested on 18
settings, our results show that ZCoT, when augmented with instructor-provided
correct answers and rubrics, produces grades that are more aligned with those
assigned by instructors compared to peer grading. Finally, our findings
indicate a promising potential for automated grading systems in MOOCs,
especially in subjects with well-defined rubrics, to improve the learning
experience for millions of online learners worldwide.

使用大型语言模型测试了三个 MOOC，发现通过零样本思维链（ZCoT）引导学生作答可以更准确地替代同伴评分，为大规模在线课程提供改进学习体验的自动评分系统。

利用大型语言模型对海量在线开放课程进行分级

Grading Massive Open Online Courses Using Large Language Models

Rubrics are a commonly used tool for labeling voice corpora in speech quality
assessment, although their application in the context of pathological speech
remains relatively limited. In this study, we introduce a comprehensive rubric
based on various dimensions of speech quality, including phonetics, fluency,
and prosody. The objective is to establish standardized criteria for
identifying errors within the speech of individuals with Down syndrome, thereby
enabling the development of automated assessment systems. To achieve this
objective, we utilized the Prautocal corpus. To assess the quality of
annotations using our rubric, two experiments were conducted, focusing on
phonetics and fluency. For phonetic evaluation, we employed the Goodness of
Pronunciation (GoP) metric, utilizing automatic segmentation systems and
correlating the results with evaluations conducted by a specialized speech
therapist. While the obtained correlation values were not notably high, a
positive trend was observed. In terms of fluency assessment, deep learning
models like wav2vec were used to extract audio features, and we employed an SVM
classifier trained on a corpus focused on identifying fluency issues to
categorize Prautocal corpus samples. The outcomes highlight the complexities of
evaluating such phenomena, with variability depending on the specific type of
disfluency detected.

我们在本研究中介绍了一种基于语音质量不同维度（包括音素、流利度和韵律）的综合性评分标准。通过利用 Prautocal 语料库，我们进行了两个实验，分别对音素和流利度进行评估。结果显示，评估这些现象的复杂性取决于检测到的特定类型的不流畅性，但我们观察到了积极的趋势。

一个综合的标注病理语音的准则

A Comprehensive Rubric for Annotating Pathological Speech

Scoring student-drawn models is time-consuming. Recently released GPT-4V
provides a unique opportunity to advance scientific modeling practices by
leveraging the powerful image processing capability. To test this ability
specifically for automatic scoring, we developed a method NERIF
(Notation-Enhanced Rubric Instruction for Few-shot Learning) employing
instructional note and rubrics to prompt GPT-4V to score students' drawn models
for science phenomena. We randomly selected a set of balanced data (N = 900)
that includes student-drawn models for six modeling assessment tasks. Each
model received a score from GPT-4V ranging at three levels: 'Beginning,'
'Developing,' or 'Proficient' according to scoring rubrics. GPT-4V scores were
compared with human experts' scores to calculate scoring accuracy. Results show
that GPT-4V's average scoring accuracy was mean =.51, SD = .037. Specifically,
average scoring accuracy was .64 for the 'Beginning' class, .62 for the
'Developing' class, and .26 for the 'Proficient' class, indicating that more
proficient models are more challenging to score. Further qualitative study
reveals how GPT-4V retrieves information from image input, including problem
context, example evaluations provided by human coders, and students' drawing
models. We also uncovered how GPT-4V catches the characteristics of
student-drawn models and narrates them in natural language. At last, we
demonstrated how GPT-4V assigns scores to student-drawn models according to the
given scoring rubric and instructional notes. Our findings suggest that the
NERIF is an effective approach for employing GPT-4V to score drawn models. Even
though there is space for GPT-4V to improve scoring accuracy, some mis-assigned
scores seemed interpretable to experts. The results of this study show that
utilizing GPT-4V for automatic scoring of student-drawn models is promising.

利用 GPT-4V 对学生绘制的模型进行自动评分的研究结果表明，尽管还有提高评分准确性的空间，但其中一些错误的评分在专家看来仍具有可解释性，这表明利用 GPT-4V 进行学生绘制模型的自动评分具有潜力。