Generative large language models (LLMs) can be a powerful tool for augmenting
text annotation procedures, but their performance varies across annotation
tasks due to prompt quality, text data idiosyncrasies, and conceptual
difficulty. Because these challenges will persist even as LLM technology
improves, we argue that any automated annotation process using an LLM must
validate the LLM's performance against labels generated by humans. To this end,
we outline a workflow to harness the annotation potential of LLMs in a
principled, efficient way. Using GPT-4, we validate this approach by
replicating 27 annotation tasks across 11 datasets from recent social science
articles in high-impact journals. We find that LLM performance for text
annotation is promising but highly contingent on both the dataset and the type
of annotation task, which reinforces the necessity to validate on a
task-by-task basis. We make available easy-to-use software designed to
implement our workflow and streamline the deployment of LLMs for automated
annotation.

通过验证人类生成的标签，我们提出了一种工作流程，以有效利用 LLM 的注释潜力，然后使用 GPT-4 复制 27 个注释任务，发现文本注释的 LLM 性能很有前途，但高度依赖数据集和注释任务类型，从而强调了必须逐任务验证的必要性。我们提供易于使用的软件，用于实现我们的工作流程，以便自动化注释的 LLM 部署。

基于生成式人工智能的自动标注需要验证

Automated Annotation with Generative AI Requires Validation

Training deep-learning-based vision systems requires the manual annotation of
a significant amount of data to optimize several parameters of the deep
convolutional neural networks. Such manual annotation is highly time-consuming
and labor-intensive. To reduce this burden, a previous study presented a fully
automated annotation approach that does not require any manual intervention.
The proposed method associates a visual marker with an object and captures it
in the same image. However, because the previous method relied on moving the
object within the capturing range using a fixed-point camera, the collected
image dataset was limited in terms of capturing viewpoints. To overcome this
limitation, this study presents a mobile application-based free-viewpoint
image-capturing method. With the proposed application, users can collect
multi-view image datasets automatically that are annotated with bounding boxes
by moving the camera. However, capturing images through human involvement is
laborious and monotonous. Therefore, we propose gamified application features
to track the progress of the collection status. Our experiments demonstrated
that using the gamified mobile application for bounding box annotation, with
visible collection progress status, can motivate users to collect multi-view
object image datasets with less mental workload and time pressure in an
enjoyable manner, leading to increased engagement.

提出了一种通过移动相机进行捕捉的自动化注释方法，并通过游戏化应用程序特性来捕获多视图图像数据集，降低了人们的心理负担和时间压力，从而增加了参与度。