High-quality distractors are crucial to both the assessment and pedagogical
value of multiple-choice questions (MCQs), where manually crafting ones that
anticipate knowledge deficiencies or misconceptions among real students is
difficult. Meanwhile, automated distractor generation, even with the help of
large language models (LLMs), remains challenging for subjects like math. It is
crucial to not only identify plausible distractors but also understand the
error behind them. In this paper, we introduce DiVERT (Distractor Generation
with Variational Errors Represented as Text), a novel variational approach that
learns an interpretable representation of errors behind distractors in math
MCQs. Through experiments on a real-world math MCQ dataset with 1,434 questions
used by hundreds of thousands of students, we show that DiVERT, despite using a
base open-source LLM with 7B parameters, outperforms state-of-the-art
approaches using GPT-4o on downstream distractor generation. We also conduct a
human evaluation with math educators and find that DiVERT leads to error labels
that are of comparable quality to human-authored ones.

利用 DiVERT 方法，本文提出一种新颖的变分方法，学习数学多项选择题中错误选项背后可解释的表示，并通过实验证明，尽管使用了拥有 7B 参数的开源大型语言模型作为基础，DiVERT 在下游错误选项生成中表现优于使用 GPT-4o 的最先进方法。同时，与数学教育专家进行的人工评估也发现，DiVERT 生成的错误标签与人工撰写的标签具有相当的质量。

DiVERT: 使用代表文本的变量错误生成干扰项的数学多项选择题

DiVERT: Distractor Generation with Variational Errors Represented as  Text for Math Multiple-choice Questions

Multiple choice questions (MCQs) are a popular method for evaluating
students' knowledge due to their efficiency in administration and grading.
Crafting high-quality math MCQs is a labor-intensive process that requires
educators to formulate precise stems and plausible distractors. Recent advances
in large language models (LLMs) have sparked interest in automating MCQ
creation, but challenges persist in ensuring mathematical accuracy and
addressing student errors. This paper introduces a prototype tool designed to
facilitate collaboration between LLMs and educators for streamlining the math
MCQ generation process. We conduct a pilot study involving math educators to
investigate how the tool can help them simplify the process of crafting
high-quality math MCQs. We found that while LLMs can generate well-formulated
question stems, their ability to generate distractors that capture common
student errors and misconceptions is limited. Nevertheless, a human-AI
collaboration has the potential to enhance the efficiency and effectiveness of
MCQ generation.

本文介绍了一个原型工具，旨在促进语言模型与教育工作者的合作，以简化数学多项选择题生成过程。我们进行了一项试点研究，涉及数学教育工作者，以调查该工具如何帮助他们简化高质量的数学多项选择题的制作过程。研究发现，虽然语言模型能够生成良好形式的问题提示，但其生成的干扰项无法涵盖常见的学生错误和误解。尽管如此，人工智能与人类的合作有潜力提高多项选择题生成的效率和有效性。

数学多项选择题生成：人类 - 大型语言模型协同

Math Multiple Choice Question Generation via Human-Large Language Model  Collaboration

Multiple-choice questions (MCQs) are ubiquitous in almost all levels of
education since they are easy to administer, grade, and are a reliable format
in both assessments and practices. An important aspect of MCQs is the
distractors, i.e., incorrect options that are designed to target specific
misconceptions or insufficient knowledge among students. To date, the task of
crafting high-quality distractors has largely remained a labor-intensive
process for teachers and learning content designers, which has limited
scalability. In this work, we explore the task of automated distractor and
corresponding feedback message generation in math MCQs using large language
models. We establish a formulation of these two tasks and propose a simple,
in-context learning-based solution. Moreover, we explore using two non-standard
metrics to evaluate the quality of the generated distractors and feedback
messages. We conduct extensive experiments on these tasks using a real-world
MCQ dataset that contains student response information. Our findings suggest
that there is a lot of room for improvement in automated distractor and
feedback generation. We also outline several directions for future work

使用大型语言模型，本研究探索了数学多项选择题中自动生成干扰项和相应反馈信息的任务，并通过非标准指标对生成的干扰项和反馈信息进行了评估，结果显示自动化干扰项和反馈生成方面仍有提升空间，并提出了几个未来研究的方向。