Ever since the emergence of large language models (LLMs) and related
applications, such as ChatGPT, its performance and error analysis for
programming tasks have been subject to research. In this work-in-progress
paper, we explore the potential of such LLMs for computing educators and
learners, as we analyze the feedback it generates to a given input containing
program code. In particular, we aim at (1) exploring how an LLM like ChatGPT
responds to students seeking help with their introductory programming tasks,
and (2) identifying feedback types in its responses. To achieve these goals, we
used students' programming sequences from a dataset gathered within a CS1
course as input for ChatGPT along with questions required to elicit feedback
and correct solutions. The results show that ChatGPT performs reasonably well
for some of the introductory programming tasks and student errors, which means
that students can potentially benefit. However, educators should provide
guidance on how to use the provided feedback, as it can contain misleading
information for novices.

探索大型语言模型在计算机教育和学习中的潜力，通过分析其对带有程序代码的输入生成的反馈进行研究，以此为目标来帮助学生解决编程任务并识别不同类型的反馈。结果表明，大型语言模型在一些入门编程任务和学生错误方面表现出了合理的性能，但教育者应提供指导，因为其提供的反馈可能对初学者包含误导性信息。

探索大型语言模型生成形成性编程反馈的潜力

Exploring the Potential of Large Language Models to Generate Formative  Programming Feedback

This paper investigates the performance of the Large Language Models (LLMs)
ChatGPT-3.5 and GPT-4 in solving introductory programming tasks. Based on the
performance, implications for didactic scenarios and assessment formats
utilizing LLMs are derived. For the analysis, 72 Python tasks for novice
programmers were selected from the free site CodingBat. Full task descriptions
were used as input to the LLMs, while the generated replies were evaluated
using CodingBat's unit tests. In addition, the general availability of textual
explanations and program code was analyzed. The results show high scores of
94.4 to 95.8% correct responses and reliable availability of textual
explanations and program code, which opens new ways to incorporate LLMs into
programming education and assessment.

该论文研究了大型语言模型（LLMs）ChatGPT-3.5 和 GPT-4 在解决入门级编程任务中的表现，并根据表现得出了利用 LLMs 进行教学场景和评估格式的暗示。研究选取了来自免费网站 CodingBat 的 72 个针对初学者的 Python 任务，使用完整任务描述作为 LLMs 的输入，通过 CodingBat 的单元测试评估生成的回复。此外，还分析了文本解释和程序代码的普遍可用性。结果显示得分高，正确响应率为 94.4％至 95.8％，同时文本解释和程序代码的可用性可靠，从而为将 LLMs 纳入编程教育和评估中打开了新的途径。