Question generation (QGen) models are often evaluated with standardized NLG metrics that are based on n-gram overlap. In this paper, we measure whether these metric improvements translate to gains in a practical setting, focusing on the use case of helping teachers automate the generation of reading comprehension quizzes. In our study, teachers building a quiz receive question suggestions, which they can either accept or refuse with a reason. Even though we find that recent progress in QGen leads to a significant increase in question acceptance rates, there is still large room for improvement, with the best model having only 68.4% of its questions accepted by the ten teachers who participated in our study. We then leverage the annotations we collected to analyze standard NLG metrics and find that model performance has reached projected upper-bounds, suggesting new automatic metrics are needed to guide QGen research forward.

本篇研究旨在利用标准化的NLG指标来检测QGen模型所带来的实际效果，并以教师自动生成阅读理解测试为例进行实际应用测试。虽然我们发现近期QGen取得了显著进展，但最佳模型仅得到了10名教师中的68.4％同意接受的问题，同时也发现需要新的自动度量标准来指导QGen研究前进。

测验设计任务：帮助教师使用自动化题目生成器创建测验