Data annotation is the labeling or tagging of raw data with relevant
information, essential for improving the efficacy of machine learning models.
The process, however, is labor-intensive and expensive. The emergence of
advanced Large Language Models (LLMs), exemplified by GPT-4, presents an
unprecedented opportunity to revolutionize and automate the intricate process
of data annotation. While existing surveys have extensively covered LLM
architecture, training, and general applications, this paper uniquely focuses
on their specific utility for data annotation. This survey contributes to three
core aspects: LLM-Based Data Annotation, Assessing LLM-generated Annotations,
and Learning with LLM-generated annotations. Furthermore, the paper includes an
in-depth taxonomy of methodologies employing LLMs for data annotation, a
comprehensive review of learning strategies for models incorporating
LLM-generated annotations, and a detailed discussion on primary challenges and
limitations associated with using LLMs for data annotation. As a key guide,
this survey aims to direct researchers and practitioners in exploring the
potential of the latest LLMs for data annotation, fostering future advancements
in this critical domain. We provide a comprehensive papers list at
https://github.com/Zhen-Tan-dmml/LLM4Annotation.git.

数据标注是提高机器学习模型效果的标记或标签化原始数据的过程，使用大型语言模型（LLMs）可以革新和自动化数据标注过程。本研究着重于 LLM 在数据标注中的特定用途，探讨了 LLM 基于数据标注、评估 LLM 生成的标注以及使用 LLM 生成的标注进行学习的方法。此外，还提供了 LLM 在数据标注中的方法学分类、包含 LLM 生成标注的模型的学习策略综述，并详细讨论了使用 LLM 进行数据标注所面临的主要挑战和限制。本文旨在为研究人员和从业者指导利用最新的 LLMs 进行数据标注，推动该关键领域的未来发展。

数据标注的大型语言模型调查

Large Language Models for Data Annotation: A Survey

Large Language Models (LLMs) have ushered in a new era of text annotation, as
their ease-of-use, high accuracy, and relatively low costs have meant that
their use has exploded in recent months. However, the rapid growth of the field
has meant that LLM-based annotation has become something of an academic Wild
West: the lack of established practices and standards has led to concerns about
the quality and validity of research. Researchers have warned that the
ostensible simplicity of LLMs can be misleading, as they are prone to bias,
misunderstandings, and unreliable results. Recognizing the transformative
potential of LLMs, this paper proposes a comprehensive set of standards and
best practices for their reliable, reproducible, and ethical use. These
guidelines span critical areas such as model selection, prompt engineering,
structured prompting, prompt stability analysis, rigorous model validation, and
the consideration of ethical and legal implications. The paper emphasizes the
need for a structured, directed, and formalized approach to using LLMs, aiming
to ensure the integrity and robustness of text annotation practices, and
advocates for a nuanced and critical engagement with LLMs in social scientific
research.

本文提出了对大型语言模型（LLMs）的可靠、可重复和符合伦理的使用的全面标准和最佳实践，涵盖了模型选择、提示工程、结构化提示、提示稳定性分析、严格的模型验证以及伦理和法律影响等关键领域，强调了对 LLMs 的结构化、有导向性和格式化的使用的需求，以确保文本注释实践的完整性和鲁棒性，并倡导在社会科学研究中对 LLMs 进行细致而批判性的参与。