The aggregation and denoising of crowd labeled data is a task that has gained increased significance with the advent of crowdsourcing platforms and massive datasets. In this paper, we propose a permutation-based model for crowd labeled data that is a significant generalization of the common Dawid-Skene model, and introduce a new error metric by which to compare different estimators. Working in a high-dimensional non-asymptotic framework that allows both the number of workers and tasks to scale, we derive optimal rates of convergence for the permutation-based model. We show that the permutation-based model offers significant robustness in estimation due to its richness, while surprisingly incurring only a small additional statistical penalty as compared to the Dawid-Skene model. Finally, we propose a computationally-efficient method, called the OBI-WAN estimator, that is uniformly optimal over a class intermediate between the permutation-based and the Dawid-Skene models, and is uniformly consistent over the entire permutation-based model class. In contrast, the guarantees for estimators available in prior literature are sub-optimal over the original Dawid-Skene model.

本文提出一种排列模型-用于聚合和去噪众包标注数据，该模型是对经典的Dawid-Skene模型的重要推广，并介绍了一种新的误差度量方法，用于比较不同的估计器。我们设计了两种计算有效的估计器：WAN估计器和OBI-WAN估计器，并对它们的性能进行了非渐近界限的界定。最后通过合成模拟和实验数据验证了理论结果。

基于置换的众包标注模型：最优估计与鲁棒性