This study explores the application of Large Language Models (LLMs),
specifically GPT-4, in the analysis of classroom dialogue, a crucial research
task for both teaching diagnosis and quality improvement. Recognizing the
knowledge-intensive and labor-intensive nature of traditional qualitative
methods in educational research, this study investigates the potential of LLM
to streamline and enhance the analysis process. The study involves datasets
from a middle school, encompassing classroom dialogues across mathematics and
Chinese classes. These dialogues were manually coded by educational experts and
then analyzed using a customised GPT-4 model. This study focuses on comparing
manual annotations with the outputs of GPT-4 to evaluate its efficacy in
analyzing educational dialogues. Time efficiency, inter-coder agreement, and
inter-coder reliability between human coders and GPT-4 are evaluated. Results
indicate substantial time savings with GPT-4, and a high degree of consistency
in coding between the model and human coders, with some discrepancies in
specific codes. These findings highlight the strong potential of LLM in
teaching evaluation and facilitation.

该研究探讨了大型语言模型（LLM），特别是 GPT-4，在课堂对话分析中的应用，这是教学诊断和质量改进的关键研究任务。研究发现传统定性方法在教育研究中具有知识密集和劳动密集的特点，调查了 LLM 在简化和增强分析过程方面的潜力。通过对中学的数据集进行分析，包括数学和语文课堂的对话，该研究对人工编码的对话进行了评估，并使用定制的 GPT-4 模型进行了分析。该研究重点比较了手动注释和 GPT-4 输出，以评估其在教育对话分析中的有效性。评估了 GPT-4 的时间效率、编码者间一致性和编码者间可靠性。结果表明，使用 GPT-4 能够显著节省时间，并且模型与人工编码者之间具有高度的一致性，尽管在某些代码上存在差异。这些发现凸显了 LLM 在教学评估和促进方面的巨大潜力。