Qualitative coding, or content analysis, extracts meaning from text to
discern quantitative patterns across a corpus of texts. Recently, advances in
the interpretive abilities of large language models (LLMs) offer potential for
automating the coding process (applying category labels to texts), thereby
enabling human researchers to concentrate on more creative research aspects,
while delegating these interpretive tasks to AI. Our case study comprises a set
of socio-historical codes on dense, paragraph-long passages representative of a
humanistic study. We show that GPT-4 is capable of human-equivalent
interpretations, whereas GPT-3.5 is not. Compared to our human-derived gold
standard, GPT-4 delivers excellent intercoder reliability (Cohen's $\kappa \geq
0.79$) for 3 of 9 codes, and substantial reliability ($\kappa \geq 0.6$) for 8
of 9 codes. In contrast, GPT-3.5 greatly underperforms for all codes
($mean(\kappa) = 0.34$; $max(\kappa) = 0.55$). Importantly, we find that coding
fidelity improves considerably when the LLM is prompted to give rationale
justifying its coding decisions (chain-of-thought reasoning). We present these
and other findings along with a set of best practices for adapting traditional
codebooks for LLMs. Our results indicate that for certain codebooks,
state-of-the-art LLMs are already adept at large-scale content analysis.
Furthermore, they suggest the next generation of models will likely render AI
coding a viable option for a majority of codebooks.

定性编码，或内容分析，从文本中提取含义，以识别文本语料库中的定量模式。最近，大型语言模型（LLM）在解释能力方面的进展为自动编码过程（对文本应用类别标签）提供了可能性，从而使人类研究人员能够集中精力进行更有创意的研究，而将这些解释性任务委托给人工智能。我们的案例研究是关于一项人文研究的一组密集段落长的社会历史代码。我们表明 GPT-4 能够提供与人类相当的解释，而 GPT-3.5 则不能。与我们基于人工导出的黄金标准相比，GPT-4 对于 9 个代码中的 3 个代码提供了出色的编码一致性（Cohen's Kappa >= 0.79），对于 8 个代码提供了显著的一致性（Kappa >= 0.6）。相比之下，GPT-3.5 在所有代码中表现不佳（mean (Kappa) = 0.34; max (Kappa) = 0.55）。重要的是，我们发现当要求 LLM 给出理由来解释其编码决策时（思路连贯推理），编码准确性显著提高。我们提出了这些以及其他发现以及一套适应 LLM 的传统编码手册的最佳实践。我们的结果表明，对于某些编码手册，最先进的 LLM 已经熟练掌握了大规模内容分析技术。此外，他们也表明，下一代模型很可能使人工智能编码成为大多数编码手册的可行选择。

LLM 在一些解释性任务中实现与人类表现相匹配的问题链条推理的可扩展性定性编码

Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning  Matches Human Performance in Some Hermeneutic Tasks

Social botnets have become an important phenomenon on social media. There are
many ways in which social bots can disrupt or influence online discourse, such
as, spam hashtags, scam twitter users, and astroturfing. In this paper we
considered one specific social botnet in Twitter to understand how it grows
over time, how the content of tweets by the social botnet differ from regular
users in the same dataset, and lastly, how the social botnet may have
influenced the relevant discussions. Our analysis is based on a qualitative
coding for approximately 3000 tweets in Arabic and English from the Syrian
social bot that was active for 35 weeks on Twitter before it was shutdown. We
find that the growth, behavior and content of this particular botnet did not
specifically align with common conceptions of botnets. Further we identify
interesting aspects of the botnet that distinguish it from regular users.

本文研究了一种特定的 Twitter 社交机器人网络，分析了其增长、内容和行为，以及该机器人网络对相关讨论的影响。通过对约 3000 条推文进行定性编码分析，发现该机器人网络的增长、行为和内容并不符合通常机器人网络的概念，并识别了与普通用户不同的有趣方面。