For the purpose of efficient and cost-effective large-scale data labeling,
crowdsourcing is increasingly being utilized. To guarantee the quality of data
labeling, multiple annotations need to be collected for each data sample, and
truth inference algorithms have been developed to accurately infer the true
labels. Despite previous studies having released public datasets to evaluate
the efficacy of truth inference algorithms, these have typically focused on a
single type of crowdsourcing task and neglected the temporal information
associated with workers' annotation activities. These limitations significantly
restrict the practical applicability of these algorithms, particularly in the
context of long-term and online truth inference. In this paper, we introduce a
substantial crowdsourcing annotation dataset collected from a real-world
crowdsourcing platform. This dataset comprises approximately two thousand
workers, one million tasks, and six million annotations. The data was gathered
over a period of approximately six months from various types of tasks, and the
timestamps of each annotation were preserved. We analyze the characteristics of
the dataset from multiple perspectives and evaluate the effectiveness of
several representative truth inference algorithms on this dataset. We
anticipate that this dataset will stimulate future research on tracking
workers' abilities over time in relation to different types of tasks, as well
as enhancing online truth inference.

通过收集真实世界众包平台上的大量数据样本，本文分析了数据集的特征并评估了几种代表性真实标签推断算法的有效性，为跟踪不同类型任务下工作者的能力随时间变化以及增强在线真实标签推断提供了启示。