Large Language Models (LLMs) have demonstrated impressive potential to simulate human behavior. Using a causal inference framework, we empirically and theoretically analyze the challenges of conducting LLM-simulated experiments, and explore potential solutions. In the context of demand estimation, we show that variations in the treatment included in the prompt (e.g., price of focal product) can cause variations in unspecified confounding factors (e.g., price of competitors, historical prices, outside temperature), introducing endogeneity and yielding implausibly flat demand curves. We propose a theoretical framework suggesting this endogeneity issue generalizes to other contexts and won't be fully resolved by merely improving the training data. Unlike real experiments where researchers assign pre-existing units across conditions, LLMs simulate units based on the entire prompt, which includes the description of the treatment. Therefore, due to associations in the training data, the characteristics of individuals and environments simulated by the LLM can be affected by the treatment assignment. We explore two potential solutions. The first specifies all contextual variables that affect both treatment and outcome, which we demonstrate to be challenging for a general-purpose LLM. The second explicitly specifies the source of treatment variation in the prompt given to the LLM (e.g., by informing the LLM that the store is running an experiment). While this approach only allows the estimation of a conditional average treatment effect that depends on the specific experimental design, it provides valuable directional results for exploratory analysis.

使用因果推断框架，我们在实证和理论上分析了进行LLM模拟实验的挑战，并探索了潜在解决方案，其中需求估计的上下文中，我们表明提示中所包含的处理变体（例如，商品价格）可能导致未指定混淆因素的变化（例如，竞争对手的价格，历史价格，外部温度），从而引入内生性并产生不太可能的平坦需求曲线。我们提出了一个理论框架，表明这种内生性问题推广到其他情境，并不能仅通过改进训练数据来完全解决。

利用语言模型模拟人类行为的挑战：因果推断视角