This study investigates the application of large language models (LLMs),
specifically GPT-3.5 and GPT-4, with Chain-of-Though (CoT)in the automatic
scoring of student-written responses to science assessments. We focused on
overcoming the challenges of accessibility, technical complexity, and lack of
explainability that have previously limited the use of automatic assessment
tools among researchers and educators. We used a testing dataset comprising six
assessment tasks (three binomial and three trinomial) with 1,650 student
responses. We employed six prompt engineering strategies, combining zero-shot
or few-shot learning with CoT, either alone or alongside item stem and scoring
rubrics. Results indicated that few-shot (acc = .67) outperformed zero-shot
learning (acc = .60), with 12.6\% increase. CoT, when used without item stem
and scoring rubrics, did not significantly affect scoring accuracy (acc = .60).
However, CoT prompting paired with contextual item stems and rubrics proved to
be a significant contributor to scoring accuracy (13.44\% increase for
zero-shot; 3.7\% increase for few-shot). Using a novel approach PPEAS, we found
a more balanced accuracy across different proficiency categories, highlighting
the importance of domain-specific reasoning in enhancing the effectiveness of
LLMs in scoring tasks. Additionally, we also found that GPT-4 demonstrated
superior performance over GPT-3.5 in various scoring tasks, showing 8.64\%
difference. The study revealed that the single-call strategy with GPT-4,
particularly using greedy sampling, outperformed other approaches, including
ensemble voting strategies. This study demonstrates the potential of LLMs in
facilitating automatic scoring, emphasizing that CoT enhances accuracy,
particularly when used with item stem and scoring rubrics.

该研究通过应用大型语言模型（LLMs），特别是 GPT-3.5 和 GPT-4，结合思维链（CoT），对学生科学评估中的写作回答进行自动评分的应用进行了调查，并专注于克服先前限制研究人员和教育工作者使用自动评估工具的可访问性、技术复杂性和解释性方面的挑战。研究结果表明，GPT-4 相对于 GPT-3.5 在各种评分任务中表现出更高的性能，并且使用 CoT 能够提高评分准确性，特别是当与项目描述和评分标准一起使用时。

应用大型语言模型和思维链路实现自动评分

Applying Large Language Models and Chain-of-Thought for Automatic  Scoring

Scoring student-drawn models is time-consuming. Recently released GPT-4V
provides a unique opportunity to advance scientific modeling practices by
leveraging the powerful image processing capability. To test this ability
specifically for automatic scoring, we developed a method NERIF
(Notation-Enhanced Rubric Instruction for Few-shot Learning) employing
instructional note and rubrics to prompt GPT-4V to score students' drawn models
for science phenomena. We randomly selected a set of balanced data (N = 900)
that includes student-drawn models for six modeling assessment tasks. Each
model received a score from GPT-4V ranging at three levels: 'Beginning,'
'Developing,' or 'Proficient' according to scoring rubrics. GPT-4V scores were
compared with human experts' scores to calculate scoring accuracy. Results show
that GPT-4V's average scoring accuracy was mean =.51, SD = .037. Specifically,
average scoring accuracy was .64 for the 'Beginning' class, .62 for the
'Developing' class, and .26 for the 'Proficient' class, indicating that more
proficient models are more challenging to score. Further qualitative study
reveals how GPT-4V retrieves information from image input, including problem
context, example evaluations provided by human coders, and students' drawing
models. We also uncovered how GPT-4V catches the characteristics of
student-drawn models and narrates them in natural language. At last, we
demonstrated how GPT-4V assigns scores to student-drawn models according to the
given scoring rubric and instructional notes. Our findings suggest that the
NERIF is an effective approach for employing GPT-4V to score drawn models. Even
though there is space for GPT-4V to improve scoring accuracy, some mis-assigned
scores seemed interpretable to experts. The results of this study show that
utilizing GPT-4V for automatic scoring of student-drawn models is promising.

利用 GPT-4V 对学生绘制的模型进行自动评分的研究结果表明，尽管还有提高评分准确性的空间，但其中一些错误的评分在专家看来仍具有可解释性，这表明利用 GPT-4V 进行学生绘制模型的自动评分具有潜力。