Critique, as a natural language description for assessing the quality of
model-generated content, has been proven to play an essential role in the
training, evaluation, and refinement of Large Language Models (LLMs). However,
there is a lack of principled understanding in evaluating the quality of the
critique itself. In this paper, we pioneer the critique of critique, termed
MetaCritique, which is a framework to evaluate the critique from two aspects,
i.e., factuality as precision score and comprehensiveness as recall score. We
calculate the harmonic mean of precision and recall as the overall rating
called F1 score. To obtain a reliable evaluation outcome, we propose Atomic
Information Units (AIUs), which describe the critique in a more fine-grained
manner. MetaCritique takes each AIU into account and aggregates each AIU's
judgment for the overall score. Moreover, given the evaluation process involves
intricate reasoning, our MetaCritique provides a natural language rationale to
support each judgment. We construct a meta-evaluation dataset containing 300
critiques (2653 AIUs) across four tasks (question answering, reasoning,
entailment, and summarization), and we conduct a comparative study to
demonstrate the feasibility and effectiveness. Experiments also show superior
critique judged by MetaCritique leads to better refinement, indicating
generative artificial intelligence indeed has the potential to be significantly
advanced with our MetaCritique. We will release relevant code and
meta-evaluation datasets at this https URL

本文中，我们开创性地提出了 MetaCritique 框架来评估评论的质量，从新颖角度对评论进行批评，通过精确度和召回率评估评论的真实性和完整性，最终采用 F1 分数来作为总体评分。我们还提出了原子信息单元 (AIUs) 用于更细致地描述评论，并对每个 AIU 的判断进行综合计算。通过构建一个包含四个任务（问题回答、推理、蕴涵和总结）的元评估数据集，我们进行了比较研究以证明 MetaCritique 的可行性和有效性。实验证明，由 MetaCritique 评估的优秀评论可以促进更好的改进，表明生成性人工智能确实有潜力与我们的 MetaCritique 显著提高。我们将在此 https URL 发布相关代码和元评估数据集。