Ever since Large Language Models (LLMs) and related applications have become
broadly available, several studies investigated their potential for assisting
educators and supporting students in higher education. LLMs such as Codex,
GPT-3.5, and GPT 4 have shown promising results in the context of large
programming courses, where students can benefit from feedback and hints if
provided timely and at scale. This paper explores the quality of GPT-4 Turbo's
generated output for prompts containing both the programming task specification
and a student's submission as input. Two assignments from an introductory
programming course were selected, and GPT-4 was asked to generate feedback for
55 randomly chosen, authentic student programming submissions. The output was
qualitatively analyzed regarding correctness, personalization, fault
localization, and other features identified in the material. Compared to prior
work and analyses of GPT-3.5, GPT-4 Turbo shows notable improvements. For
example, the output is more structured and consistent. GPT-4 Turbo can also
accurately identify invalid casing in student programs' output. In some cases,
the feedback also includes the output of the student program. At the same time,
inconsistent feedback was noted such as stating that the submission is correct
but an error needs to be fixed. The present work increases our understanding of
LLMs' potential, limitations, and how to integrate them into e-assessment
systems, pedagogical scenarios, and instructing students who are using
applications based on GPT-4.

该研究探索了使用 GPT-4 Turbo 生成学生编程提交的反馈，研究结果显示相比 GPT-3.5，GPT-4 Turbo 在输出质量上有显著的改进，可以更准确地识别学生程序输出中的错误，并能为学生提供结构化和一致的反馈。此研究还增加了我们对大型语言模型在电子评估系统、教学场景和指导学生使用基于 GPT-4 的应用中潜力和限制的理解。