Identifying scientific publications that are within a dynamic field of
research often requires costly annotation by subject-matter experts. Resources
like widely-accepted classification criteria or field taxonomies are
unavailable for a domain like artificial intelligence (AI), which spans
emerging topics and technologies. We address these challenges by inferring a
functional definition of AI research from existing expert labels, and then
evaluating state-of-the-art chatbot models on the task of expert data
annotation. Using the arXiv publication database as ground-truth, we experiment
with prompt engineering for GPT chatbot models to identify an alternative,
automated expert annotation pipeline that assigns AI labels with 94% accuracy.
For comparison, we fine-tune SPECTER, a transformer language model pre-trained
on scientific publications, that achieves 96% accuracy (only 2% higher than
GPT) on classifying AI publications. Our results indicate that with effective
prompt engineering, chatbots can be used as reliable data annotators even where
subject-area expertise is required. To evaluate the utility of
chatbot-annotated datasets on downstream classification tasks, we train a new
classifier on GPT-labeled data and compare its performance to the arXiv-trained
model. The classifier trained on GPT-labeled data outperforms the arXiv-trained
model by nine percentage points, achieving 82% accuracy.

使用 GPT 聊天机器人模型进行有效的提示工程，可以将聊天机器人用作可靠的数据标注工具，从而实现基于人工智能的科学出版物的自动标注，其准确率达到 94%。用 GPT 标注的数据训练的分类器在性能上优于 arXiv 训练的模型，达到 82% 的准确率。

AI 中的 AI：探索 GPT 作为 AI 出版物专家注释工具的实用性

AI on AI: Exploring the Utility of GPT as an Expert Annotator of AI  Publications

Large Language Models (LLMs) like ChatGPT are capable of successfully
performing many language processing tasks zero-shot (without the need for
training data). If this capacity also applies to the coding of social phenomena
like persuasiveness and political ideology, then LLMs could effectively
transform Computational Social Science (CSS). This work provides a road map for
using LLMs as CSS tools. Towards this end, we contribute a set of prompting
best practices and an extensive evaluation pipeline to measure the zero-shot
performance of 13 language models on 24 representative CSS benchmarks. On
taxonomic labeling tasks (classification), LLMs fail to outperform the best
fine-tuned models but still achieve fair levels of agreement with humans. On
free-form coding tasks (generation), LLMs produce explanations that often
exceed the quality of crowdworkers' gold references. We conclude that today's
LLMs can radically augment the CSS research pipeline in two ways: (1) serving
as zero-shot data annotators on human annotation teams, and (2) bootstrapping
challenging creative generation tasks (e.g., explaining the hidden meaning
behind text). In summary, LLMs can significantly reduce costs and increase
efficiency of social science analysis in partnership with humans.

本研究探讨了利用大型语言模型 (Large Language Models，LLMs) 作为计算社会科学 (Computational Social Science，CSS) 工具的可行性。结果表明，LLMs 可以作为零 - shot 数据注释工具或是用于启动具有挑战性的生成任务，并可以极大地提高社会科学分析的效率和降低成本。