Online programming videos, including tutorials and streamcasts, are widely
popular and contain a wealth of expert knowledge. However, effectively
utilizing these resources to achieve targeted learning goals can be
challenging. Unlike direct tutoring, video content lacks tailored guidance
based on individual learning paces, personalized feedback, and interactive
engagement necessary for support and monitoring. Our work transforms
programming videos into one-on-one tutoring experiences using the cognitive
apprenticeship framework. Tutorly, developed as a JupyterLab Plugin, allows
learners to (1) set personalized learning goals, (2) engage in
learning-by-doing through a conversational LLM-based mentor agent, (3) receive
guidance and feedback based on a student model that steers the mentor moves. In
a within-subject study with 16 participants learning exploratory data analysis
from a streamcast, Tutorly significantly improved their performance from 61.9%
to 76.6% based on a post-test questionnaire. Tutorly demonstrates the potential
for enhancing programming video learning experiences with LLM and learner
modeling.

我们的研究将编程视频转化为个性化的一对一辅导经验，通过认知学徒框架，学习者可以制定个性化学习目标，通过基于学生模型的对话式 LLM 导师代理进行学以致用，并根据学生模型提供指导和反馈。一项针对 16 名参与者的实验表明，在从一个实时流传单中学习探索性数据分析的过程中，Tutorly 使他们的表现从 61.9％提高到 76.6％。这项研究显示了 LLM 和学习者建模在增强编程视频学习体验方面的潜力。

Tutorly：利用 LLMs 将编程视频转化为学徒式学习环境

Tutorly: Turning Programming Videos Into Apprenticeship Learning  Environments with LLMs

One-on-one tutoring is widely acknowledged as an effective instructional
method, conditioned on qualified tutors. However, the high demand for qualified
tutors remains a challenge, often necessitating the training of novice tutors
(i.e., trainees) to ensure effective tutoring. Research suggests that providing
timely explanatory feedback can facilitate the training process for trainees.
However, it presents challenges due to the time-consuming nature of assessing
trainee performance by human experts. Inspired by the recent advancements of
large language models (LLMs), our study employed the GPT-4 model to build an
explanatory feedback system. This system identifies trainees' responses in
binary form (i.e., correct/incorrect) and automatically provides template-based
feedback with responses appropriately rephrased by the GPT-4 model. We
conducted our study on 410 responses from trainees across three training
lessons: Giving Effective Praise, Reacting to Errors, and Determining What
Students Know. Our findings indicate that: 1) using a few-shot approach, the
GPT-4 model effectively identifies correct/incorrect trainees' responses from
three training lessons with an average F1 score of 0.84 and an AUC score of
0.85; and 2) using the few-shot approach, the GPT-4 model adeptly rephrases
incorrect trainees' responses into desired responses, achieving performance
comparable to that of human experts.

通过使用 GPT-4 模型，我们的研究建立一个解释性反馈系统，用于训练初级导师。研究结果表明，该模型有效地识别出三个训练课程中正确 / 错误的学员反馈，并将错误反馈转化为期望的反馈，其性能达到与人类专家相当的水平。

如何做到对？使用 GPT 重述错误学员回答

How Can I Get It Right? Using GPT to Rephrase Incorrect Trainee  Responses

One-on-one tutoring is an effective instructional method for enhancing
learning, yet its efficacy hinges on tutor competencies. Novice math tutors
often prioritize content-specific guidance, neglecting aspects such as
social-emotional learning. Social-emotional learning promotes equity and
inclusion and nurturing relationships with students, which is crucial for
holistic student development. Assessing the competencies of tutors accurately
and efficiently can drive the development of tailored tutor training programs.
However, evaluating novice tutor ability during real-time tutoring remains
challenging as it typically requires experts-in-the-loop. To address this
challenge, this preliminary study aims to harness Generative Pre-trained
Transformers (GPT), such as GPT-3.5 and GPT-4 models, to automatically assess
tutors' ability of using social-emotional tutoring strategies. Moreover, this
study also reports on the financial dimensions and considerations of employing
these models in real-time and at scale for automated assessment. The current
study examined four prompting strategies: two basic Zero-shot prompt
strategies, Tree of Thought prompt, and Retrieval-Augmented Generator (RAG)
based prompt. The results indicate that the RAG prompt demonstrated more
accurate performance (assessed by the level of hallucination and correctness in
the generated assessment texts) and lower financial costs than the other
strategies evaluated. These findings inform the development of personalized
tutor training interventions to enhance the the educational effectiveness of
tutored learning.

通过使用 GPT 模型自动评估教师使用社交情感辅导策略的能力，研究发现 RAG 提示方式比其他评估策略具有更准确的表现和更低的财务成本，这为个性化教师培训干预的发展提供了指导。