Manually annotating data for computational social science tasks can be
costly, time-consuming, and emotionally draining. While recent work suggests
that LLMs can perform such annotation tasks in zero-shot settings, little is
known about how prompt design impacts LLMs' compliance and accuracy. We conduct
a large-scale multi-prompt experiment to test how model selection (ChatGPT,
PaLM2, and Falcon7b) and prompt design features (definition inclusion, output
type, explanation, and prompt length) impact the compliance and accuracy of
LLM-generated annotations on four CSS tasks (toxicity, sentiment, rumor stance,
and news frames). Our results show that LLM compliance and accuracy are highly
prompt-dependent. For instance, prompting for numerical scores instead of
labels reduces all LLMs' compliance and accuracy. The overall best prompting
setup is task-dependent, and minor prompt changes can cause large changes in
the distribution of generated labels. By showing that prompt design
significantly impacts the quality and distribution of LLM-generated
annotations, this work serves as both a warning and practical guide for
researchers and practitioners.

手动注释计算社会科学任务的数据成本高昂、耗时且情感压力大。最近的研究表明，零 - shot 设置下，语言模型可以执行此类注释任务，但我们对于提示设计如何影响语言模型的遵从和准确性了解甚少。我们进行了大规模的多提示实验，以测试模型选择（ChatGPT、PaLM2 和 Falcon7b）和提示设计特征（定义包含、输出类型、解释和提示长度）对 LLM 生成注释的遵从和准确性的影响，针对四个计算社会科学任务（毒性、情感、谣言态度和新闻框架）。我们的结果表明，LLM 的遵从和准确性高度依赖于提示。例如，提示使用数值得分而不是标签会降低所有 LLM 的遵从和准确性。整体而言，最佳的提示设置取决于任务，微小的提示更改会导致生成标签分布上的巨大变化。通过显示提示设计对 LLM 生成注释的质量和分布有显著影响，本研究作为研究人员和从业者的警示和实践指南。