Reference-based metrics such as BLEU and BERTScore are widely used to
evaluate question generation (QG). In this study, on QG benchmarks such as
SQuAD and HotpotQA, we find that using human-written references cannot
guarantee the effectiveness of the reference-based metrics. Most QG benchmarks
have only one reference; we replicated the annotation process and collect
another reference. A good metric was expected to grade a human-validated
question no worse than generated questions. However, the results of
reference-based metrics on our newly collected reference disproved the metrics
themselves. We propose a reference-free metric consisted of multi-dimensional
criteria such as naturalness, answerability, and complexity, utilizing large
language models. These criteria are not constrained to the syntactic or
semantic of a single reference question, and the metric does not require a
diverse set of references. Experiments reveal that our metric accurately
distinguishes between high-quality questions and flawed ones, and achieves
state-of-the-art alignment with human judgment.

该研究在使用 BLEU 和 BERTScore 等基于参考文献的指标评估问题生成（QG）时发现，使用人工编写的参考文献不能保证参考文献指标的有效性。作者提出了一种基于自由参考文献的多维度标准的度量方法，通过利用大型语言模型来评估问题的自然性、可回答性和复杂性，该度量方法不受单个参考问题的句法或语义限制，也不需要多样化的参考文献集合。实验证明，该度量方法能够准确区分高质量问题和有缺陷的问题，并与人类判断达到最先进的一致性。