Large language models (LLMs) are trained to imitate humans to explain human
decisions. However, do LLMs explain themselves? Can they help humans build
mental models of how LLMs process different inputs? To answer these questions,
we propose to evaluate $\textbf{counterfactual simulatability}$ of natural
language explanations: whether an explanation can enable humans to precisely
infer the model's outputs on diverse counterfactuals of the explained input.
For example, if a model answers "yes" to the input question "Can eagles fly?"
with the explanation "all birds can fly", then humans would infer from the
explanation that it would also answer "yes" to the counterfactual input "Can
penguins fly?". If the explanation is precise, then the model's answer should
match humans' expectations.
We implemented two metrics based on counterfactual simulatability: precision
and generality. We generated diverse counterfactuals automatically using LLMs.
We then used these metrics to evaluate state-of-the-art LLMs (e.g., GPT-4) on
two tasks: multi-hop factual reasoning and reward modeling. We found that LLM's
explanations have low precision and that precision does not correlate with
plausibility. Therefore, naively optimizing human approvals (e.g., RLHF) may
not be a sufficient solution.

本篇论文中，我们提出了一种新的自然语言解释评估方法 —— 反事实模拟性。实验结果显示，当前大型语言模型 (LLMs) 的解释精度相对较低，精度与可信度相关性也不高。因此，单纯依赖人类评判可能不足以解决问题。