Large Language Models (LLMs) like ChatGPT are capable of successfully
performing many language processing tasks zero-shot (without the need for
training data). If this capacity also applies to the coding of social phenomena
like persuasiveness and political ideology, then LLMs could effectively
transform Computational Social Science (CSS). This work provides a road map for
using LLMs as CSS tools. Towards this end, we contribute a set of prompting
best practices and an extensive evaluation pipeline to measure the zero-shot
performance of 13 language models on 24 representative CSS benchmarks. On
taxonomic labeling tasks (classification), LLMs fail to outperform the best
fine-tuned models but still achieve fair levels of agreement with humans. On
free-form coding tasks (generation), LLMs produce explanations that often
exceed the quality of crowdworkers' gold references. We conclude that today's
LLMs can radically augment the CSS research pipeline in two ways: (1) serving
as zero-shot data annotators on human annotation teams, and (2) bootstrapping
challenging creative generation tasks (e.g., explaining the hidden meaning
behind text). In summary, LLMs can significantly reduce costs and increase
efficiency of social science analysis in partnership with humans.

本研究探讨了利用大型语言模型 (Large Language Models，LLMs) 作为计算社会科学 (Computational Social Science，CSS) 工具的可行性。结果表明，LLMs 可以作为零 - shot 数据注释工具或是用于启动具有挑战性的生成任务，并可以极大地提高社会科学分析的效率和降低成本。