Causal inference studies using textual social media data can provide
actionable insights on human behavior. Making accurate causal inferences with
text requires controlling for confounding which could otherwise impart bias.
Recently, many different methods for adjusting for confounders have been
proposed, and we show that these existing methods disagree with one another on
two datasets inspired by previous social media studies. Evaluating causal
methods is challenging, as ground truth counterfactuals are almost never
available. Presently, no empirical evaluation framework for causal methods
using text exists, and as such, practitioners must select their methods without
guidance. We contribute the first such framework, which consists of five tasks
drawn from real world studies. Our framework enables the evaluation of any
casual inference method using text. Across 648 experiments and two datasets, we
evaluate every commonly used causal inference method and identify their
strengths and weaknesses to inform social media researchers seeking to use such
methods, and guide future improvements. We make all tasks, data, and models
public to inform applications and encourage additional research.

该研究提供了第一个结合真实世界研究任务的文本因果推理方法的经验评估框架，评估了常用文本因果推理方法的优缺点，并公开任务、数据和模型，以促进应用和鼓励额外研究。