Large Language Models (LLMs) such as ChatGPT have demonstrated remarkable
performance across various tasks and have garnered significant attention from
both researchers and practitioners. However, in an educational context, we
still observe a performance gap in generating distractors -- i.e., plausible
yet incorrect answers -- with LLMs for multiple-choice questions (MCQs). In
this study, we propose a strategy for guiding LLMs such as ChatGPT, in
generating relevant distractors by prompting them with question items
automatically retrieved from a question bank as well-chosen in-context
examples. We evaluate our LLM-based solutions using a quantitative assessment
on an existing test set, as well as through quality annotations by human
experts, i.e., teachers. We found that on average 53% of the generated
distractors presented to the teachers were rated as high-quality, i.e.,
suitable for immediate use as is, outperforming the state-of-the-art model. We
also show the gains of our approach 1 in generating high-quality distractors by
comparing it with a zero-shot ChatGPT and a few-shot ChatGPT prompted with
static examples.

通过从题库自动检索题目并将其用作合适的上下文示例，本研究提出了一种引导 ChatGPT 等大型语言模型生成相关干扰项的策略，通过对现有测试集进行数量评估和教师质量评注，我们发现教师评价中有 53% 的生成干扰项被评为高质量，即可立即使用，优于最先进的模型，与零样本 ChatGPT 和少样本 ChatGPT 通过静态示例进行比较，显示了我们方法在生成高质量干扰项方面的优势。