Critique, as a natural language description for assessing the quality of
model-generated content, has been proven to play an essential role in the
training, evaluation, and refinement of Large Language Models (LLMs). However,
there is a lack of principled understanding in evaluating the quality of the
critique itself. In this paper, we pioneer the critique of critique, termed
MetaCritique, which is a framework to evaluate the critique from two aspects,
i.e., factuality as precision score and comprehensiveness as recall score. We
calculate the harmonic mean of precision and recall as the overall rating
called F1 score. To obtain a reliable evaluation outcome, we propose Atomic
Information Units (AIUs), which describe the critique in a more fine-grained
manner. MetaCritique takes each AIU into account and aggregates each AIU's
judgment for the overall score. Moreover, given the evaluation process involves
intricate reasoning, our MetaCritique provides a natural language rationale to
support each judgment. We construct a meta-evaluation dataset containing 300
critiques (2653 AIUs) across four tasks (question answering, reasoning,
entailment, and summarization), and we conduct a comparative study to
demonstrate the feasibility and effectiveness. Experiments also show superior
critique judged by MetaCritique leads to better refinement, indicating
generative artificial intelligence indeed has the potential to be significantly
advanced with our MetaCritique. We will release relevant code and
meta-evaluation datasets at this https URL

本文中，我们开创性地提出了 MetaCritique 框架来评估评论的质量，从新颖角度对评论进行批评，通过精确度和召回率评估评论的真实性和完整性，最终采用 F1 分数来作为总体评分。我们还提出了原子信息单元 (AIUs) 用于更细致地描述评论，并对每个 AIU 的判断进行综合计算。通过构建一个包含四个任务（问题回答、推理、蕴涵和总结）的元评估数据集，我们进行了比较研究以证明 MetaCritique 的可行性和有效性。实验证明，由 MetaCritique 评估的优秀评论可以促进更好的改进，表明生成性人工智能确实有潜力与我们的 MetaCritique 显著提高。我们将在此 https URL 发布相关代码和元评估数据集。

批评的批评

The Critique of Critique

We present a novel platform for evaluating the capability of Large Language
Models (LLMs) to autonomously compose and critique survey papers spanning a
vast array of disciplines including sciences, humanities, education, and law.
Within this framework, AI systems undertake a simulated peer-review mechanism
akin to traditional scholarly journals, with human organizers serving in an
editorial oversight capacity. Within this framework, we organized a competition
for the AutoML conference 2023. Entrants are tasked with presenting stand-alone
models adept at authoring articles from designated prompts and subsequently
appraising them. Assessment criteria include clarity, reference
appropriateness, accountability, and the substantive value of the content. This
paper presents the design of the competition, including the implementation
baseline submissions and methods of evaluation.

我们提出了一种新颖的平台，用于评估大型语言模型（LLMs）自主撰写和评论跨科学、人文、教育和法律等各个学科的调研论文的能力。这个框架中，人工智能系统通过模拟同行评审机制进行操作，类似于传统学术期刊，人类组织者担任编辑监督职责。在这个框架内，我们为 2023 年 AutoML 会议组织了一次竞赛。参赛者的任务是根据指定提示撰写独立的文章，并对其进行评估。评估标准包括清晰度、参考文献适当性、责任性和内容的实质价值。本文介绍了竞赛的设计，包括实施基线提交和评估方法。

自动调查挑战

Auto-survey Challenge

In a recent workshop paper, Massiceti et al. presented a baseline model and
subsequent critique of Visual Dialog (Das et al., CVPR 2017) that raises what
we believe to be unfounded concerns about the dataset and evaluation. This
article intends to rebut the critique and clarify potential confusions for
practitioners and future participants in the Visual Dialog challenge.

本文回应了 Massiceti 等人对 CVPR2017 Visual Dialog 论文基准模型和数据集评估提出的批评，澄清了可能会使从业者和未来参与 Visual Dialog 挑战者感到困惑的问题。