We evaluate an automatic hint generator for CS1 programming assignments
powered by GPT-4, a large language model. This system provides natural language
guidance about how students can improve their incorrect solutions to short
programming exercises. A hint can be requested each time a student fails a test
case. Our evaluation addresses three Research Questions:
RQ1: Do the hints help students improve their code? RQ2: How effectively do
the hints capture problems in student code? RQ3: Are the issues that students
resolve the same as the issues addressed in the hints?
To address these research questions quantitatively, we identified a set of
fine-grained knowledge components and determined which ones apply to each
exercise, incorrect solution, and generated hint. Comparing data from two large
CS1 offerings, we found that access to the hints helps students to address
problems with their code more quickly, that hints are able to consistently
capture the most pressing errors in students' code, and that hints that address
a few issues at once rather than a single bug are more likely to lead to direct
student progress.

通过 GPT-4 大型语言模型提供的自动提示生成器，我们评估了一种针对 CS1 编程作业的系统，该系统提供关于学生如何改进错误解决方案的自然语言指导。我们回答了三个研究问题：RQ1. 提示是否帮助学生改进代码？RQ2. 提示在捕捉学生代码问题方面的效果如何？RQ3. 学生解决的问题与提示中提到的问题是否相同？通过定量分析，我们确定了一组细粒度的知识组件，并确定了每个练习、错误解决方案和生成的提示适用哪些组件。通过对两个大型的 CS1 课程数据进行对比，我们发现，访问提示有助于学生更快地解决代码问题，提示能够一致地捕捉到学生代码中最紧迫的错误，并且同时处理几个问题而不是单个 bug 的提示更有可能直接帮助学生取得进展。