Multiple-choice questions with item-writing flaws can negatively impact
student learning and skew analytics. These flaws are often present in
student-generated questions, making it difficult to assess their quality and
suitability for classroom usage. Existing methods for evaluating
multiple-choice questions often focus on machine readability metrics, without
considering their intended use within course materials and their pedagogical
implications. In this study, we compared the performance of a rule-based method
we developed to a machine-learning based method utilizing GPT-4 for the task of
automatically assessing multiple-choice questions based on 19 common
item-writing flaws. By analyzing 200 student-generated questions from four
different subject areas, we found that the rule-based method correctly detected
91% of the flaws identified by human annotators, as compared to 79% by GPT-4.
We demonstrated the effectiveness of the two methods in identifying common
item-writing flaws present in the student-generated questions across different
subject areas. The rule-based method can accurately and efficiently evaluate
multiple-choice questions from multiple domains, outperforming GPT-4 and going
beyond existing metrics that do not account for the educational use of such
questions. Finally, we discuss the potential for using these automated methods
to improve the quality of questions based on the identified flaws.

本研究采用基于规则和基于 GPT-4 的基于机器学习方法评估了 200 个学生生成的多项选择题目中 19 个常见选项撰写错误。结果显示基于规则的方法可以更准确，更高效地评估多个学科领域的选择题，并在教育使用方面具有潜在校准的作用。