Automatically generated questions often suffer from problems such as unclear
expression or factual inaccuracies, requiring a reliable and comprehensive
evaluation of their quality. Human evaluation is frequently used in the field
of question generation (QG) and is one of the most accurate evaluation methods.
It also serves as the standard for automatic metrics. However, there is a lack
of unified evaluation criteria, which hampers the development of both QG
technologies and automatic evaluation methods. To address this, we propose
QGEval, a multi-dimensional Evaluation benchmark for Question Generation, which
evaluates both generated questions and existing automatic metrics across 7
dimensions: fluency, clarity, conciseness, relevance, consistency,
answerability, and answer consistency. We demonstrate the appropriateness of
these dimensions by examining their correlations and distinctions. Analysis
with QGEval reveals that 1) most QG models perform unsatisfactorily in terms of
answerability and answer consistency, and 2) existing metrics fail to align
well with human assessments when evaluating generated questions across the 7
dimensions. We expect this work to foster the development of both QG
technologies and automatic metrics for QG.

我们提出了一个名为 QGEval 的多维度评估标准，用于对生成的问题和现有的自动评估方法进行评估，涵盖了流畅度、清晰度、简洁度、相关性、一致性、可回答性和回答一致性等七个维度。通过 QGEval 的分析，我们发现大多数问题生成模型在可回答性和回答一致性方面表现不尽人意，并且现有的评估指标无法很好地与人类评估结果相吻合。我们希望这项工作能促进问题生成技术和问题生成自动评估方法的发展。