Annotated data plays a critical role in Natural Language Processing (NLP) in
training models and evaluating their performance. Given recent developments in
Large Language Models (LLMs), models such as ChatGPT demonstrate zero-shot
capability on many text-annotation tasks, comparable with or even exceeding
human annotators. Such LLMs can serve as alternatives for manual annotation,
due to lower costs and higher scalability. However, limited work has leveraged
LLMs as complementary annotators, nor explored how annotation work is best
allocated among humans and LLMs to achieve both quality and cost objectives. We
propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of
unstructured texts at scale. Under this framework, we utilize uncertainty to
estimate LLMs' annotation capability. Our empirical study shows CoAnnotating to
be an effective means to allocate work from results on different datasets, with
up to 21% performance improvement over random baseline. For code
implementation, see this https URL

使用大规模语言模型和人工标注相结合的新范式 CoAnnotating，在未标注的文本上进行人工标注和语言模型标注，以达到更好的质量和成本效益。