Free-text rationales play a pivotal role in explainable NLP, bridging the
knowledge and reasoning gaps behind a model's decision-making. However, due to
the diversity of potential reasoning paths and a corresponding lack of
definitive ground truth, their evaluation remains a challenge. Existing
evaluation metrics rely on the degree to which a rationale supports a target
label, but we find these fall short in evaluating rationales that inadvertently
leak the labels. To address this problem, we propose RORA, a Robust free-text
Rationale evaluation against label leakage. RORA quantifies the new information
supplied by a rationale to justify the label. This is achieved by assessing the
conditional V-information \citep{hewitt-etal-2021-conditional} with a
predictive family robust against leaky features that can be exploited by a
small model. RORA consistently outperforms existing approaches in evaluating
human-written, synthetic, or model-generated rationales, particularly
demonstrating robustness against label leakage. We also show that RORA aligns
well with human judgment, providing a more reliable and accurate measurement
across diverse free-text rationales.

通过评估条件性 V - 信息以及对容易被小型模型利用的敏感特征进行鲁棒性评估，我们提出了一种针对标签泄漏的强韧的自由文本基于理由的评估方法 RORA，它在评估人类编写、合成或模型生成的基于理由时提供了更可靠和准确的度量，尤其是对于标签泄漏展现出了鲁棒性。

RORA: 强健自由文本理由评估

RORA: Robust Free-Text Rationale Evaluation

Recent work on explainable NLP has shown that few-shot prompting can enable
large pretrained language models (LLMs) to generate grammatical and factual
natural language explanations for data labels. In this work, we study the
connection between explainability and sample hardness by investigating the
following research question - "Are LLMs and humans equally good at explaining
data labels for both easy and hard samples?" We answer this question by first
collecting human-written explanations in the form of generalizable commonsense
rules on the task of Winograd Schema Challenge (Winogrande dataset). We compare
these explanations with those generated by GPT-3 while varying the hardness of
the test samples as well as the in-context samples. We observe that (1) GPT-3
explanations are as grammatical as human explanations regardless of the
hardness of the test samples, (2) for easy examples, GPT-3 generates highly
supportive explanations but human explanations are more generalizable, and (3)
for hard examples, human explanations are significantly better than GPT-3
explanations both in terms of label-supportiveness and generalizability
judgements. We also find that hardness of the in-context examples impacts the
quality of GPT-3 explanations. Finally, we show that the supportiveness and
generalizability aspects of human explanations are also impacted by sample
hardness, although by a much smaller margin than models. Supporting code and
data are available at this https URL

通过人类编写的常识规则与 GPT-3 生成的描述进行比较，本研究研究了 LLM、人类在简单与困难例子的数据标注解释方面的能力，研究结果表明对于简单示例，GPT-3 生成的解释具有高度的支持性，而人类解释更具有通用性，对于困难示例，人类解释在标签支持和通用判断方面都比 GPT-3 解释更好。

难例也更难解释吗？人类和模型生成的解释研究

Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations

Explainable NLP (ExNLP) has increasingly focused on collecting
human-annotated textual explanations. These explanations are used downstream in
three ways: as data augmentation to improve performance on a predictive task,
as supervision to train models to produce explanations for their predictions,
and as a ground-truth to evaluate model-generated explanations. In this review,
we identify 65 datasets with three predominant classes of textual explanations
(highlights, free-text, and structured), organize the literature on annotating
each type, identify strengths and shortcomings of existing collection
methodologies, and give recommendations for collecting ExNLP datasets in the
future.

本文总结了解释性自然语言处理领域近年来集中收集人类注释的文本解释，将其分为三类 (highlights, free-text, and structured) 并对其注释方法进行整理，提出了现有采集方法的优点和缺点，给出了未来收集数据集的建议。