Chatbots based on Large Language Models (LLMs) have shown strong capabilities
in language understanding. In this study, we explore the potential of LLMs in
assisting corpus-based linguistic studies through automatic annotation of texts
with specific categories of linguistic information. Specifically, we examined
to what extent LLMs understand the functional elements constituting the speech
act of apology from a local grammar perspective, by comparing the performance
of ChatGPT (powered by GPT-3.5), Bing chatbot (powered by GPT-4), and a human
coder in the annotation task. The results demonstrate that Bing chatbot
significantly outperformed ChatGPT in the task. Compared to human annotator,
the overall performance of Bing chatbot was slightly less satisfactory.
However, it already achieved high F1 scores: 99.95% for the tag of APOLOGISING,
91.91% for REASON, 95.35% for APOLOGISER, 89.74% for APOLOGISEE, and 96.47% for
INTENSIFIER. Therefore, we propose that LLM-assisted annotation is a promising
automated approach for corpus studies.

本研究旨在探索大型语言模型（LLMs）在语料库语言学研究中自动标注文本的潜力，证明了使用 Bing 聊天机器人进行标注的性能优于使用 ChatGPT 或人工标注，并建议 LLM 辅助标注是一种有前途的语料库研究自动化方法。

使用 LLM 辅助标注进行语料库语言学研究：本地语法分析的案例研究

Using LLM-assisted Annotation for Corpus Linguistics: A Case Study of  Local Grammar Analysis

We examine a methodology using neural language models (LMs) for analyzing the
word order of language. This LM-based method has the potential to overcome the
difficulties existing methods face, such as the propagation of preprocessor
errors in count-based methods. In this study, we explore whether the LM-based
method is valid for analyzing the word order. As a case study, this study
focuses on Japanese due to its complex and flexible word order. To validate the
LM-based method, we test (i) parallels between LMs and human word order
preference, and (ii) consistency of the results obtained using the LM-based
method with previous linguistic studies. Through our experiments, we
tentatively conclude that LMs display sufficient word order knowledge for usage
as an analysis tool. Finally, using the LM-based method, we demonstrate the
relationship between the canonical word order and topicalization, which had yet
to be analyzed by large-scale experiments.

本文介绍了一种使用神经语言模型 (LMs) 分析语言词序的方法，以日本语作为案例研究，通过实验，初步得出神经语言模型具有足够的词序知识用于分析工具，并利用该方法，显示了规范词序与主题化之间的关系。