Ever since the emergence of large language models (LLMs) and related
applications, such as ChatGPT, its performance and error analysis for
programming tasks have been subject to research. In this work-in-progress
paper, we explore the potential of such LLMs for computing educators and
learners, as we analyze the feedback it generates to a given input containing
program code. In particular, we aim at (1) exploring how an LLM like ChatGPT
responds to students seeking help with their introductory programming tasks,
and (2) identifying feedback types in its responses. To achieve these goals, we
used students' programming sequences from a dataset gathered within a CS1
course as input for ChatGPT along with questions required to elicit feedback
and correct solutions. The results show that ChatGPT performs reasonably well
for some of the introductory programming tasks and student errors, which means
that students can potentially benefit. However, educators should provide
guidance on how to use the provided feedback, as it can contain misleading
information for novices.

探索大型语言模型在计算机教育和学习中的潜力，通过分析其对带有程序代码的输入生成的反馈进行研究，以此为目标来帮助学生解决编程任务并识别不同类型的反馈。结果表明，大型语言模型在一些入门编程任务和学生错误方面表现出了合理的性能，但教育者应提供指导，因为其提供的反馈可能对初学者包含误导性信息。

探索大型语言模型生成形成性编程反馈的潜力

Exploring the Potential of Large Language Models to Generate Formative  Programming Feedback

To use reinforcement learning from human feedback (RLHF) in practical
applications, it is crucial to learn reward models from diverse sources of
human feedback and to consider human factors involved in providing feedback of
different types. However, the systematic study of learning from diverse types
of feedback is held back by limited standardized tooling available to
researchers. To bridge this gap, we propose RLHF-Blender, a configurable,
interactive interface for learning from human feedback. RLHF-Blender provides a
modular experimentation framework and implementation that enables researchers
to systematically investigate the properties and qualities of human feedback
for reward learning. The system facilitates the exploration of various feedback
types, including demonstrations, rankings, comparisons, and natural language
instructions, as well as studies considering the impact of human factors on
their effectiveness. We discuss a set of concrete research opportunities
enabled by RLHF-Blender. More information is available at
this https URL

使用来自不同来源的人类反馈學习强化学习模型，RLHF-Blender 是一个可配置的，互动式界面，帮助研究人员系统地研究人类反馈的属性和质量，以及人类因素对其有效性的影响。