Large language models (LLMs) are remarkable data annotators. They can be used
to generate high-fidelity supervised training data, as well as survey and
experimental data. With the widespread adoption of LLMs, human gold--standard
annotations are key to understanding the capabilities of LLMs and the validity
of their results. However, crowdsourcing, an important, inexpensive way to
obtain human annotations, may itself be impacted by LLMs, as crowd workers have
financial incentives to use LLMs to increase their productivity and income. To
investigate this concern, we conducted a case study on the prevalence of LLM
usage by crowd workers. We reran an abstract summarization task from the
literature on Amazon Mechanical Turk and, through a combination of keystroke
detection and synthetic text classification, estimate that 33-46% of crowd
workers used LLMs when completing the task. Although generalization to other,
less LLM-friendly tasks is unclear, our results call for platforms,
researchers, and crowd workers to find new ways to ensure that human data
remain human, perhaps using the methodology proposed here as a stepping stone.
Code/data: this https URL

本文研究 LLMs 对众包工人的影响，通过对在 Amazon Mechanical Turk 上运行的一个摘要任务，发现 33-46% 的众包工人使用了 LLMs，因此需要找到新的方法确保数据来源于人类本身。

人工人工人工智能：众包工作者广泛使用大型语言模型进行文本生成任务

Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use  Large Language Models for Text Production Tasks

Recent studies show that Question Answering (QA) based on Answer Sentence
Selection (AS2) can be improved by generating an improved answer from the top-k
ranked answer sentences (termed GenQA). This allows for synthesizing the
information from multiple candidates into a concise, natural-sounding answer.
However, creating large-scale supervised training data for GenQA models is very
challenging. In this paper, we propose to train a GenQA model by transferring
knowledge from a trained AS2 model, to overcome the aforementioned issue.
First, we use an AS2 model to produce a ranking over answer candidates for a
set of questions. Then, we use the top ranked candidate as the generation
target, and the next k top ranked candidates as context for training a GenQA
model. We also propose to use the AS2 model prediction scores for loss
weighting and score-conditioned input/output shaping, to aid the knowledge
transfer. Our evaluation on three public and one large industrial datasets
demonstrates the superiority of our approach over the AS2 baseline, and GenQA
trained using supervised data.

本文提出一种通过从 AS2 模型中传递知识来训练 GenQA 模型的方法，该模型将候选答案进行排序，使用排名最高的答案作为生成目标，并使用前 k 个排名最高的答案作为上下文进行训练。模型还利用 AS2 模型的预测得分进行损失权重和条件输入 / 输出塑形，以协助知识传递，并在三个公共数据集和一个大型工业数据集上的评估中证明了该方法优于 AS2 基线和使用监督数据训练的 GenQA。