Deductive coding is a widely used qualitative research method for determining
the prevalence of themes across documents. While useful, deductive coding is
often burdensome and time consuming since it requires researchers to read,
interpret, and reliably categorize a large body of unstructured text documents.
Large language models (LLMs), like ChatGPT, are a class of quickly evolving AI
tools that can perform a range of natural language processing and reasoning
tasks. In this study, we explore the use of LLMs to reduce the time it takes
for deductive coding while retaining the flexibility of a traditional content
analysis. We outline the proposed approach, called LLM-assisted content
analysis (LACA), along with an in-depth case study using GPT-3.5 for LACA on a
publicly available deductive coding data set. Additionally, we conduct an
empirical benchmark using LACA on 4 publicly available data sets to assess the
broader question of how well GPT-3.5 performs across a range of deductive
coding tasks. Overall, we find that GPT-3.5 can often perform deductive coding
at levels of agreement comparable to human coders. Additionally, we demonstrate
that LACA can help refine prompts for deductive coding, identify codes for
which an LLM is randomly guessing, and help assess when to use LLMs vs. human
coders for deductive coding. We conclude with several implications for future
practice of deductive coding and related research methods.

本研究探讨了利用大型语言模型的方法来减少演绎编码所需的时间，同时保留传统内容分析的灵活性。研究发现，在某些情况下，使用 GPT-3.5 可以达到与人类编码者同等程度的一致性水平。

LLM 辅助内容分析：使用大型语言模型支持演绎编码

LLM-Assisted Content Analysis: Using Large Language Models to Support  Deductive Coding

Qualitative analysis of textual contents unpacks rich and valuable
information by assigning labels to the data. However, this process is often
labor-intensive, particularly when working with large datasets. While recent
AI-based tools demonstrate utility, researchers may not have readily available
AI resources and expertise, let alone be challenged by the limited
generalizability of those task-specific models. In this study, we explored the
use of large language models (LLMs) in supporting deductive coding, a major
category of qualitative analysis where researchers use pre-determined codebooks
to label the data into a fixed set of codes. Instead of training task-specific
models, a pre-trained LLM could be used directly for various tasks without
fine-tuning through prompt learning. Using a curiosity-driven questions coding
task as a case study, we found, by combining GPT-3 with expert-drafted
codebooks, our proposed approach achieved fair to substantial agreements with
expert-coded results. We lay out challenges and opportunities in using LLMs to
support qualitative coding and beyond.

本文研究了大型语言模型（LLMs）在支持定性分析中的应用，通过结合 GPT-3 和专家编写的编码手册，我们提出的方法在好奇驱动的问题编码任务中达到了与专家编码结果相当的一致性。