The automatic generation of Multiple Choice Questions (MCQ) has the potential
to reduce the time educators spend on student assessment significantly.
However, existing evaluation metrics for MCQ generation, such as BLEU, ROUGE,
and METEOR, focus on the n-gram based similarity of the generated MCQ to the
gold sample in the dataset and disregard their educational value. They fail to
evaluate the MCQ's ability to assess the student's knowledge of the
corresponding target fact. To tackle this issue, we propose a novel automatic
evaluation metric, coined Knowledge Dependent Answerability (KDA), which
measures the MCQ's answerability given knowledge of the target fact.
Specifically, we first show how to measure KDA based on student responses from
a human survey. Then, we propose two automatic evaluation metrics, KDA_disc and
KDA_cont, that approximate KDA by leveraging pre-trained language models to
imitate students' problem-solving behavior. Through our human studies, we show
that KDA_disc and KDA_soft have strong correlations with both (1) KDA and (2)
usability in an actual classroom setting, labeled by experts. Furthermore, when
combined with n-gram based similarity metrics, KDA_disc and KDA_cont are shown
to have a strong predictive power for various expert-labeled MCQ quality
measures.

该研究提出了一种新的自动评估指标，称为知识相关可回答性（KDA），该指标可以衡量多项选择题的可回答程度，提高了现有的评估模型，使其不仅基于生成题目与样本的相似度，而且考虑了评估题目对应知识点能力的能力，从而可以更好地评估评估题目。