Numerous decision-making tasks require estimating causal effects under
interventions on different parts of a system. As practitioners consider using
large language models (LLMs) to automate decisions, studying their causal
reasoning capabilities becomes crucial. A recent line of work evaluates LLMs
ability to retrieve commonsense causal facts, but these evaluations do not
sufficiently assess how LLMs reason about interventions. Motivated by the role
that interventions play in causal inference, in this paper, we conduct
empirical analyses to evaluate whether LLMs can accurately update their
knowledge of a data-generating process in response to an intervention. We
create benchmarks that span diverse causal graphs (e.g., confounding,
mediation) and variable types, and enable a study of intervention-based
reasoning. These benchmarks allow us to isolate the ability of LLMs to
accurately predict changes resulting from their ability to memorize facts or
find other shortcuts. Our analysis on four LLMs highlights that while GPT- 4
models show promising accuracy at predicting the intervention effects, they
remain sensitive to distracting factors in the prompts.

评估大型语言模型在干预作用下准确更新其对数据生成过程的知识的能力，以及对因果推断中不同因果图和变量类型的干预性推理的研究。研究结果表明，虽然 GPT-4 模型在预测干预效果方面表现出有希望的准确性，但它们对提示中的干扰因素仍然敏感。