Large language models (LLMs) have shown great potential for the automatic
generation of feedback in a wide range of computing contexts. However, concerns
have been voiced around the privacy and ethical implications of sending student
work to proprietary models. This has sparked considerable interest in the use
of open source LLMs in education, but the quality of the feedback that such
open models can produce remains understudied. This is a concern as providing
flawed or misleading generated feedback could be detrimental to student
learning. Inspired by recent work that has utilised very powerful LLMs, such as
GPT-4, to evaluate the outputs produced by less powerful models, we conduct an
automated analysis of the quality of the feedback produced by several open
source models using a dataset from an introductory programming course. First,
we investigate the viability of employing GPT-4 as an automated evaluator by
comparing its evaluations with those of a human expert. We observe that GPT-4
demonstrates a bias toward positively rating feedback while exhibiting moderate
agreement with human raters, showcasing its potential as a feedback evaluator.
Second, we explore the quality of feedback generated by several leading
open-source LLMs by using GPT-4 to evaluate the feedback. We find that some
models offer competitive performance with popular proprietary LLMs, such as
ChatGPT, indicating opportunities for their responsible use in educational
settings.

通过使用 GPT-4 对多个开源语言模型的反馈进行评估，本研究探讨了大型语言模型在教育领域中生成反馈的质量，并发现一些模型在性能上可以与专有的语言模型竞争，从而为其在教学环境中的负责任使用提供机会。