Machine learning-based automatic scoring can be challenging if students'
responses are unbalanced across scoring categories, as it introduces
uncertainty in the machine training process. To meet this challenge, we
introduce a novel text data augmentation framework leveraging GPT-4, a
generative large language model, specifically tailored for unbalanced datasets
in automatic scoring. Our experimental dataset comprised student written
responses to two science items. We crafted prompts for GPT-4 to generate
responses resembling student written answers, particularly for the minority
scoring classes, to augment the data. We then finetuned DistillBERT for
automatic scoring based on the augmented and original datasets. Model
performance was assessed using accuracy, precision, recall, and F1 metrics. Our
findings revealed that incorporating GPT-4-augmented data remarkedly improved
model performance, particularly for precision, recall, and F1 scores.
Interestingly, the extent of improvement varied depending on the specific
dataset and the proportion of augmented data used. Notably, we found that a
varying amount of augmented data (5\%-40\%) was needed to obtain stable
improvement for automatic scoring. We also compared the accuracies of models
trained with GPT-4 augmented data to those trained with additional
student-written responses. Results suggest that the GPT-4 augmented scoring
models outperform or match the models trained with student-written augmented
data. This research underscores the potential and effectiveness of data
augmentation techniques utilizing generative large language models--GPT-4 in
addressing unbalanced datasets within automated assessment.

通过利用 GPT-4 生成大型语言模型来进行文本数据增强，以解决在自动评分中存在的不均衡数据集的挑战。研究结果表明，GPT-4 增强评分模型在精确度、召回率和 F1 分数方面明显改善，并且对于稳定改善自动评分所需的增强数据比例存在差异。与其他学生编写的响应相比，GPT-4 增强评分模型显示出更好的性能，从而强调了利用生成型大型语言模型进行数据增强技术在自动评估中的潜力和有效性。