The advent of AI driven large language models (LLMs) have stirred discussions
about their role in qualitative research. Some view these as tools to enrich
human understanding, while others perceive them as threats to the core values
of the discipline. This study aimed to compare and contrast the comprehension
capabilities of humans and LLMs. We conducted an experiment with small sample
of Alexa app reviews, initially classified by a human analyst. LLMs were then
asked to classify these reviews and provide the reasoning behind each
classification. We compared the results with human classification and
reasoning. The research indicated a significant alignment between human and
ChatGPT 3.5 classifications in one third of cases, and a slightly lower
alignment with GPT4 in over a quarter of cases. The two AI models showed a
higher alignment, observed in more than half of the instances. However, a
consensus across all three methods was seen only in about one fifth of the
classifications. In the comparison of human and LLMs reasoning, it appears that
human analysts lean heavily on their individual experiences. As expected, LLMs,
on the other hand, base their reasoning on the specific word choices found in
app reviews and the functional components of the app itself. Our results
highlight the potential for effective human LLM collaboration, suggesting a
synergistic rather than competitive relationship. Researchers must continuously
evaluate LLMs role in their work, thereby fostering a future where AI and
humans jointly enrich qualitative research.

本研究对语言模型（LLMs）的理解能力进行了比较和对照，发现人类分析师和 LLMs 的分类和推理能力存在显著差异，但二者合作可能会产生协同效应，从而丰富了定性研究。