We evaluated the capability of generative pre-trained transformers (GPT), to pass assessments in introductory and intermediate Python programming courses at the postsecondary level. Discussions of potential uses (e.g., exercise generation, code explanation) and misuses (e.g., cheating) of this emerging technology in programming education have intensified, but to date there has not been a rigorous analysis of the models' capabilities in the realistic context of a full-fledged programming course with diverse set of assessment instruments. We evaluated GPT on three Python courses that employ assessments ranging from simple multiple-choice questions (no code involved) to complex programming projects with code bases distributed into multiple files (599 exercises overall). Further, we studied if and how successfully GPT models leverage feedback provided by an auto-grader. We found that the current models are not capable of passing the full spectrum of assessments typically involved in a Python programming course (<70% on even entry-level modules). Yet, it is clear that a straightforward application of these easily accessible models could enable a learner to obtain a non-trivial portion of the overall available score (>55%) in introductory and intermediate courses alike. While the models exhibit remarkable capabilities, including correcting solutions based on auto-grader's feedback, some limitations exist (e.g., poor handling of exercises requiring complex chains of reasoning steps). These findings can be leveraged by instructors wishing to adapt their assessments so that GPT becomes a valuable assistant for a learner as opposed to an end-to-end solution.

本文评估了GPT在大学的Python编程课程中通过评估的能力，并研究了GPT模型如何利用自动评估程序提供的反馈。研究发现，这些模型都不能完全通过Python编程课程中的各种评估，但简单应用它们可以使学习者在入门和中级课程中获得非常好的分数。然而，这些模型存在某些局限性，如对需要复杂推理步骤的练习处理能力较差。因此，本文建议教师改变评估方式，使GPT变成学习者的有价值的辅助工具，而不是全自动解决方案。

生成式预训练转换器（GPT）能否通过高等教育编程课程的评估？