Lectures are a learning experience for both students and teachers. Students
learn from teachers about the subject material, while teachers learn from
students about how to refine their instruction. However, online student
feedback is unstructured and abundant, making it challenging for teachers to
learn and improve. We take a step towards tackling this challenge. First, we
contribute a dataset for studying this problem: SIGHT is a large dataset of 288
math lecture transcripts and 15,784 comments collected from the Massachusetts
Institute of Technology OpenCourseWare (MIT OCW) YouTube channel. Second, we
develop a rubric for categorizing feedback types using qualitative analysis.
Qualitative analysis methods are powerful in uncovering domain-specific
insights, however they are costly to apply to large data sources. To overcome
this challenge, we propose a set of best practices for using large language
models (LLMs) to cheaply classify the comments at scale. We observe a striking
correlation between the model's and humans' annotation: Categories with
consistent human annotations (>$0.9$ inter-rater reliability, IRR) also display
higher human-model agreement (>$0.7$), while categories with less consistent
human annotations ($0.7$-$0.8$ IRR) correspondingly demonstrate lower
human-model agreement ($0.3$-$0.5$). These techniques uncover useful student
feedback from thousands of comments, costing around $\$0.002$ per comment. We
conclude by discussing exciting future directions on using online student
feedback and improving automated annotation techniques for qualitative
research.

本研究提供了一个大的数据集，开发了一种标准对学生的评论进行分类，并提出了一种使用大型语言模型进行评论分类的最佳实践，从而建立了一个低成本的定量研究自动化体系，来处理海量的在线学生反馈。