The emergence of large language models (LLMs) has opened up exciting
possibilities for simulating human behavior and cognitive processes, with
potential applications in various domains, including marketing research and
consumer behavior analysis. However, the validity of utilizing LLMs as
stand-ins for human subjects remains uncertain due to glaring divergences that
suggest fundamentally different underlying processes at play and the
sensitivity of LLM responses to prompt variations. This paper presents a novel
approach based on Shapley values from cooperative game theory to interpret LLM
behavior and quantify the relative contribution of each prompt component to the
model's output. Through two applications-a discrete choice experiment and an
investigation of cognitive biases-we demonstrate how the Shapley value method
can uncover what we term "token noise" effects, a phenomenon where LLM
decisions are disproportionately influenced by tokens providing minimal
informative content. This phenomenon raises concerns about the robustness and
generalizability of insights obtained from LLMs in the context of human
behavior simulation. Our model-agnostic approach extends its utility to
proprietary LLMs, providing a valuable tool for marketers and researchers to
strategically optimize prompts and mitigate apparent cognitive biases. Our
findings underscore the need for a more nuanced understanding of the factors
driving LLM responses before relying on them as substitutes for human subjects
in research settings. We emphasize the importance of researchers reporting
results conditioned on specific prompt templates and exercising caution when
drawing parallels between human behavior and LLMs.

利用合作博弈理论中的 Shapley 值独特方法解读大型语言模型的行为和量化每个提示组件对模型输出的相对贡献，揭示了被称为 “标记噪声” 效应的现象，该现象引发对大型语言模型在人类行为模拟中获得见解的健壮性和普适性的担忧。该研究强调在依赖大型语言模型作为研究对象替代品之前，需要更加细致地了解驱动大型语言模型响应的因素，并强调研究人员在报告结果时受到特定提示模板的条件限制，并要谨慎地在人类行为和大型语言模型之间绘制并行之间的类比。