Evaluating free-text explanations is a multifaceted, subjective, and
labor-intensive task. Large language models (LLMs) present an appealing
alternative due to their potential for consistency, scalability, and
cost-efficiency. In this work, we present ACORN, a new dataset of 3,500
free-text explanations and aspect-wise quality ratings, and use it to gain
insights into how LLMs evaluate explanations. We observed that replacing one of
the human ratings sometimes maintained, but more often lowered the
inter-annotator agreement across different settings and quality aspects,
suggesting that their judgments are not always consistent with human raters. We
further quantified this difference by comparing the correlation between
LLM-generated ratings with majority-voted human ratings across different
quality aspects. With the best system, Spearman's rank correlation ranged
between 0.53 to 0.95, averaging 0.72 across aspects, indicating moderately high
but imperfect alignment. Finally, we considered the alternative of using an LLM
as an additional rater when human raters are scarce, and measured the
correlation between majority-voted labels with a limited human pool and LLMs as
an additional rater, compared to the original gold labels. While GPT-4 improved
the outcome when there were only two human raters, in all other observed cases,
LLMs were neutral to detrimental when there were three or more human raters. We
publicly release the dataset to support future improvements in LLM-in-the-loop
evaluation here: this https URL

评估自由文本解释是一个多方面、主观、费时的任务。大型语言模型呈现出一种吸引人的替代方案，因为它们具有一致性、可扩展性和成本效益方面的潜力。在这项工作中，我们提出了 ACORN 数据集，其中包含 3,500 个自由文本解释和按方面的质量评分，并利用该数据集来了解语言模型如何评估解释。我们观察到，有时替换人工评分会维持，但更常见的是降低不同设置和质量方面之间的评分一致性，这表明它们的判断并不总是与人工评定者一致。我们进一步通过比较不同质量方面上 LLM 生成的评分与大多数人工评分之间的相关性来量化这种差异。在最佳系统中，Spearman 等级相关性在不同方面之间的范围为 0.53 到 0.95，平均为 0.72，表示有着中等高度但不完全一致的相关。最后，我们考虑了在人工评定者有限的情况下，使用 LLM 作为额外的评定者的替代方案，并测量了多数人工标签与有限人工队列和 LLM 作为额外评定者之间的相关性，与原始金标签进行比较。当只有两个人工评定者时，GPT-4 改善了结果，在所有其他观察到的情况中，在有三个或更多人工评定者时，LLMs 对结果中立或有害。我们公开发布此数据集以支持未来 LLM 在环评估的改进。

ACORN: 方面级常识推理解释评估

ACORN: Aspect-wise Commonsense Reasoning Explanation Evaluation

Large language models are increasingly capable of generating fluent-appearing
text with relatively little task-specific supervision. But can these models
accurately explain classification decisions? We consider the task of generating
free-text explanations using human-written examples in a few-shot manner. We
find that (1) authoring higher quality prompts results in higher quality
generations; and (2) surprisingly, in a head-to-head comparison, crowdworkers
often prefer explanations generated by GPT-3 to crowdsourced explanations in
existing datasets. Our human studies also show, however, that while models
often produce factual, grammatical, and sufficient explanations, they have room
to improve along axes such as providing novel information and supporting the
label. We create a pipeline that combines GPT-3 with a supervised filter that
incorporates binary acceptability judgments from humans in the loop. Despite
the intrinsic subjectivity of acceptability judgments, we demonstrate that
acceptability is partially correlated with various fine-grained attributes of
explanations. Our approach is able to consistently filter GPT-3-generated
explanations deemed acceptable by humans.

本文探讨利用人类书写的样例以几乎无监督的方式创作自由文本解释的任务，发现高质量的提示有助于提高语言模型的生成效果，同时人类研究表明 GPT-3 生成的解释在某些情况下能够胜过人工生成的解释；作者还结合 GPT-3 与学习自评价的筛选器对生成的解释进行过滤，结果表明这一方法能够实现较高水平的解释过滤。