Generative large language models (LLMs) can be a powerful tool for augmenting
text annotation procedures, but their performance varies across annotation
tasks due to prompt quality, text data idiosyncrasies, and conceptual
difficulty. Because these challenges will persist even as LLM technology
improves, we argue that any automated annotation process using an LLM must
validate the LLM's performance against labels generated by humans. To this end,
we outline a workflow to harness the annotation potential of LLMs in a
principled, efficient way. Using GPT-4, we validate this approach by
replicating 27 annotation tasks across 11 datasets from recent social science
articles in high-impact journals. We find that LLM performance for text
annotation is promising but highly contingent on both the dataset and the type
of annotation task, which reinforces the necessity to validate on a
task-by-task basis. We make available easy-to-use software designed to
implement our workflow and streamline the deployment of LLMs for automated
annotation.

通过验证人类生成的标签，我们提出了一种工作流程，以有效利用 LLM 的注释潜力，然后使用 GPT-4 复制 27 个注释任务，发现文本注释的 LLM 性能很有前途，但高度依赖数据集和注释任务类型，从而强调了必须逐任务验证的必要性。我们提供易于使用的软件，用于实现我们的工作流程，以便自动化注释的 LLM 部署。