Identifying logical errors in complex, incomplete or even contradictory and
overall heterogeneous data like students' experimentation protocols is
challenging. Recognizing the limitations of current evaluation methods, we
investigate the potential of Large Language Models (LLMs) for automatically
identifying student errors and streamlining teacher assessments. Our aim is to
provide a foundation for productive, personalized feedback. Using a dataset of
65 student protocols, an Artificial Intelligence (AI) system based on the
GPT-3.5 and GPT-4 series was developed and tested against human raters. Our
results indicate varying levels of accuracy in error detection between the AI
system and human raters. The AI system can accurately identify many fundamental
student errors, for instance, the AI system identifies when a student is
focusing the hypothesis not on the dependent variable but solely on an expected
observation (acc. = 0.90), when a student modifies the trials in an ongoing
investigation (acc. = 1), and whether a student is conducting valid test trials
(acc. = 0.82) reliably. The identification of other, usually more complex
errors, like whether a student conducts a valid control trial (acc. = .60),
poses a greater challenge. This research explores not only the utility of AI in
educational settings, but also contributes to the understanding of the
capabilities of LLMs in error detection in inquiry-based learning like
experimentation.

使用大型语言模型（LLMs）探索错误检测在探究式学习中的应用，并解决学生实验协议等复杂、不完整、甚至矛盾和多样化的数据中的逻辑错误识别的困难。

利用人工智能和大型语言模型评估实验中学生的错误：与人类评分员的比较研究

Assessing Student Errors in Experimentation Using Artificial  Intelligence and Large Language Models: A Comparative Study with Human Raters

This study investigates the use of Artificial Intelligence (AI)-powered,
multi-role chatbots as a means to enhance learning experiences and foster
engagement in computer science education. Leveraging a design-based research
approach, we develop, implement, and evaluate a novel learning environment
enriched with four distinct chatbot roles: Instructor Bot, Peer Bot, Career
Advising Bot, and Emotional Supporter Bot. These roles, designed around the
tenets of Self-Determination Theory, cater to the three innate psychological
needs of learners - competence, autonomy, and relatedness. Additionally, the
system embraces an inquiry-based learning paradigm, encouraging students to ask
questions, seek solutions, and explore their curiosities.
We test this system in a higher education context over a period of one month
with 200 participating students, comparing outcomes with conditions involving a
human tutor and a single chatbot. Our research utilizes a mixed-methods
approach, encompassing quantitative measures such as chat log sequence
analysis, and qualitative methods including surveys and focus group interviews.
By integrating cutting-edge Natural Language Processing techniques such as
topic modelling and sentiment analysis, we offer an in-depth understanding of
the system's impact on learner engagement, motivation, and inquiry-based
learning.
This study, through its rigorous design and innovative approach, provides
significant insights into the potential of AI-empowered, multi-role chatbots in
reshaping the landscape of computer science education and fostering an
engaging, supportive, and motivating learning environment.

该研究通过设计并评价了一种 AI 驱动的多任务聊天机器人，在计算机科学教育中提升学习体验、促进学生参与。研究采用设计研究方法，开发了一个新型学习环境，其中包括指导机器人、同伴机器人、职业规划机器人和情感支持机器人四个不同的聊天机器人角色，通过满足学生的能力、自主和关联性等三个内在心理需求，以探究为基础的学习模式鼓励学生提问、寻求解决方案和探索好奇心。研究通过在一个月内与 200 名参与学生进行高等教育环境测试，将结果与人类导师和单一聊天机器人条件进行对比。研究运用混合研究方法，包括聊天记录序列分析、调查和焦点小组访谈等定量和定性手段。通过整合先进的自然语言处理技术，如主题建模和情感分析，提供了对该系统对学习者参与度、动机和探究型学习的深入理解。该研究通过其严格的设计和创新的方法，为 AI 驱动的多任务聊天机器人在重塑计算机科学教育以及创建引人入胜、支持性和激励性学习环境的潜力提供了重要见解。