Evaluating the quality of automatically generated question items has been a
long standing challenge. In this paper, we leverage LLMs to simulate student
profiles and generate responses to multiple-choice questions (MCQs). The
generative students' responses to MCQs can further support question item
evaluation. We propose Generative Students, a prompt architecture designed
based on the KLI framework. A generative student profile is a function of the
list of knowledge components the student has mastered, has confusion about or
has no evidence of knowledge of. We instantiate the Generative Students concept
on the subject domain of heuristic evaluation. We created 45 generative
students using GPT-4 and had them respond to 20 MCQs. We found that the
generative students produced logical and believable responses that were aligned
with their profiles. We then compared the generative students' responses to
real students' responses on the same set of MCQs and found a high correlation.
Moreover, there was considerable overlap in the difficult questions identified
by generative students and real students. A subsequent case study demonstrated
that an instructor could improve question quality based on the signals provided
by Generative Students.

通过利用 LLMs 模拟学生档案并生成多项选择题的回答，本文提出了一种基于 KLI 框架设计的生成学生提示结构，该结构可用于评估问题实例的质量，并发现生成学生的回答与真实学生的回答具有高度相关性。