In order to oversee advanced AI systems, it is important to understand their
underlying decision-making process. When prompted, large language models (LLMs)
can provide natural language explanations or reasoning traces that sound
plausible and receive high ratings from human annotators. However, it is
unclear to what extent these explanations are faithful, i.e., truly capture the
factors responsible for the model's predictions. In this work, we introduce
Correlational Explanatory Faithfulness (CEF), a metric that can be used in
faithfulness tests based on input interventions. Previous metrics used in such
tests take into account only binary changes in the predictions. Our metric
accounts for the total shift in the model's predicted label distribution, more
accurately reflecting the explanations' faithfulness. We then introduce the
Correlational Counterfactual Test (CCT) by instantiating CEF on the
Counterfactual Test (CT) from Atanasova et al. (2023). We evaluate the
faithfulness of free-text explanations generated by few-shot-prompted LLMs from
the Llama2 family on three NLP tasks. We find that our metric measures aspects
of faithfulness which the CT misses.

评估了 Llama2 系列的少样本提示 LLMs 生成的自由文本解释在三个 NLP 任务上的忠实性，并发现我们的度量考虑了 CT 遗漏的忠实性方面。