Large Language Models (LLMs) are promising analytical tools. They can augment
human epistemic, cognitive and reasoning abilities, and support 'sensemaking',
making sense of a complex environment or subject by analysing large volumes of
data with a sensitivity to context and nuance absent in earlier text processing
systems. This paper presents a pilot experiment that explores how LLMs can
support thematic analysis of controversial topics. We compare how human
researchers and two LLMs GPT-4 and Llama 2 categorise excerpts from media
coverage of the controversial Australian Robodebt scandal. Our findings
highlight intriguing overlaps and variances in thematic categorisation between
human and machine agents, and suggest where LLMs can be effective in supporting
forms of discourse and thematic analysis. We argue LLMs should be used to
augment, and not replace human interpretation, and we add further
methodological insights and reflections to existing research on the application
of automation to qualitative research methods. We also introduce a novel
card-based design toolkit, for both researchers and practitioners to further
interrogate LLMs as analytical tools.

该研究试验了大型语言模型（LLMs）在支持主题分析上的作用，发现人工研究者与两个 LLMs 在对澳大利亚 Robodebt 丑闻媒体报道的片段进行主题分类时存在有趣的重叠和差异，表明 LLMs 在支持对话和主题分析方面可以发挥作用，但应用中应该作为人类解释的补充，同时该研究还介绍了一种新颖的基于卡片的设计工具箱，以进一步研究 LLMs 作为分析工具的潜力。