This study highlights the potential of fine-tuned ChatGPT (GPT-3.5) for
automatically scoring student written constructed responses using example
assessment tasks in science education. Recent studies on OpenAI's generative
model GPT-3.5 proved its superiority in predicting the natural language with
high accuracy and human-like responses. GPT-3.5 has been trained over enormous
online language materials such as journals and Wikipedia; therefore, more than
direct usage of pre-trained GPT-3.5 is required for automatic scoring as
students utilize a different language than trained material. These imply that a
domain-specific model, fine-tuned over data for specific tasks, can enhance
model performance. In this study, we fine-tuned GPT-3.5 on six assessment tasks
with a diverse dataset of middle-school and high-school student responses and
expert scoring. The six tasks comprise two multi-label and four multi-class
assessment tasks. We compare the performance of fine-tuned GPT-3.5 with the
fine-tuned state-of-the-art Google's generated language model, BERT. The
results show that in-domain training corpora constructed from science questions
and responses for BERT achieved average accuracy = 0.838, SD = 0.069. GPT-3.5
shows a remarkable average increase (9.1%) in automatic scoring accuracy (mean
= 9.15, SD = 0.042) for the six tasks, p =0.001 < 0.05. Specifically, for
multi-label tasks (item 1 with 5 labels; item 2 with 10 labels), GPT-3.5
achieved significantly higher scoring accuracy than BERT across all the labels,
with the second item achieving a 7.1% increase. The average scoring increase
for the four multi-class items for GPT-3.5 was 10.6% compared to BERT. Our
study confirmed the effectiveness of fine-tuned GPT-3.5 for automatic scoring
of student responses on domain-specific data in education with high accuracy.
We have released fine-tuned models for public use and community engagement.

本研究通过在科学教育中使用示例评估任务，突显了精调 ChatGPT（GPT-3.5）自动评分学生书面构建性答案的潜力。与经过领域特定训练数据细调的 Google 生成语言模型 BERT 相比，GPT-3.5 在自动评分准确性上表现出显著提高，并发布了细调模型以供公众使用和社区参与。