We introduce CheckEval, a novel evaluation framework using Large Language
Models, addressing the challenges of ambiguity and inconsistency in current
evaluation methods. CheckEval addresses these challenges by dividing evaluation
criteria into detailed sub-aspects and constructing a checklist of Boolean
questions for each, simplifying the evaluation. This approach not only renders
the process more interpretable but also significantly enhances the robustness
and reliability of results by focusing on specific evaluation dimensions.
Validated through a focused case study using the SummEval benchmark, CheckEval
indicates a strong correlation with human judgments. Furthermore, it
demonstrates a highly consistent Inter-Annotator Agreement. These findings
highlight the effectiveness of CheckEval for objective, flexible, and precise
evaluations. By offering a customizable and interactive framework, CheckEval
sets a new standard for the use of LLMs in evaluation, responding to the
evolving needs of the field and establishing a clear method for future
LLM-based evaluation.

CheckEval 是一个使用大型语言模型的新型评估框架，通过将评估标准细分为详细的子方面，并构建每个方面的布尔问题清单，以简化评估过程中的模棱两可性和不一致性。该方法不仅使过程更具可解释性，还通过专注于特定的评估维度，显著提高了结果的健壮性和可靠性。CheckEval 在使用 SummEval 基准进行的重点案例研究中获得验证，表明与人类判断之间存在强烈的相关性，并且表现出高度一致的标注者间一致性。这些发现突显了 CheckEval 在客观、灵活和精确评估方面的有效性。通过提供可定制和交互式的框架，CheckEval 为语言模型评估树立了新的标准，回应了该领域不断发展的需求，并为未来基于语言模型的评估建立了清晰的方法。