Common crowdsourcing systems average estimates of a latent quantity of
interest provided by many crowdworkers to produce a group estimate. We develop
a new approach -- just-predict-others -- that leverages self-supervised
learning and a novel aggregation scheme. This approach adapts weights assigned
to crowdworkers based on estimates they provided for previous quantities. When
skills vary across crowdworkers or their estimates correlate, the weighted sum
offers a more accurate group estimate than the average. Existing algorithms
such as expectation maximization can, at least in principle, produce similarly
accurate group estimates. However, their computational requirements become
onerous when complex models, such as neural networks, are required to express
relationships among crowdworkers. Just-predict-others accommodates such
complexity as well as many other practical challenges. We analyze the efficacy
of just-predict-others through theoretical and computational studies. Among
other things, we establish asymptotic optimality as the number of engagements
per crowdworker grows.

利用自监督学习和新颖的聚合机制，just-predict-others 方法在面对具有不同技能水平和相关估计的众包工作者时，比传统的众包系统更能提供准确的群体估计；并通过理论和计算研究验证 just-predict-others 的效力，证明随着每个众包工作者参与次数的增加，该方法在渐近优化方面表现优秀。

自适应众包通过自监督学习

Adaptive Crowdsourcing Via Self-Supervised Learning

Crowdsourcing systems have been used to accumulate massive amounts of labeled
data for applications such as computer vision and natural language processing.
However, because crowdsourced labeling is inherently dynamic and uncertain,
developing a technique that can work in most situations is extremely
challenging. In this paper, we introduce Crowd-Certain, a novel approach for
label aggregation in crowdsourced and ensemble learning classification tasks
that offers improved performance and computational efficiency for different
numbers of annotators and a variety of datasets. The proposed method uses the
consistency of the annotators versus a trained classifier to determine a
reliability score for each annotator. Furthermore, Crowd-Certain leverages
predicted probabilities, enabling the reuse of trained classifiers on future
sample data, thereby eliminating the need for recurrent simulation processes
inherent in existing methods. We extensively evaluated our approach against ten
existing techniques across ten different datasets, each labeled by varying
numbers of annotators. The findings demonstrate that Crowd-Certain outperforms
the existing methods (Tao, Sheng, KOS, MACE, MajorityVote, MMSR, Wawa,
Zero-Based Skill, GLAD, and Dawid Skene), in nearly all scenarios, delivering
higher average accuracy, F1 scores, and AUC rates. Additionally, we introduce a
variation of two existing confidence score measurement techniques. Finally we
evaluate these two confidence score techniques using two evaluation metrics:
Expected Calibration Error (ECE) and Brier Score Loss. Our results show that
Crowd-Certain achieves higher Brier Score, and lower ECE across the majority of
the examined datasets, suggesting better calibrated results.

Crowd-Certain 是一种在众包和集成学习分类任务中进行标签聚合的新方法，通过评估标注者的一致性和受训分类器，确定每个标注者的可靠性分数，并利用预测概率实现对未来样本数据的训练分类器的复用，大幅提升性能和计算效率。在对十个不同数据集进行了广泛评估后，Crowd-Certain 几乎在所有场景中优于现有的十种方法，提供更高的平均准确率、F1 分数和 AUC 率。另外，我们引入了两种现有置信度评估技术的变体，并通过两种评估指标（期望校准误差和 Brier 得分损失）对其进行了评估，结果表明 Crowd-Certain 在大部分数据集上具有更高的 Brier 得分和更低的校准误差，表明结果的校准更好。

众所周知：众包和集成学习分类中的标签聚合

Crowd-Certain: Label Aggregation in Crowdsourced and Ensemble Learning  Classification

Worker quality control is a crucial aspect of crowdsourcing systems;
typically occupying a large fraction of the time and money invested on
crowdsourcing. In this work, we devise techniques to generate confidence
intervals for worker error rate estimates, thereby enabling a better evaluation
of worker quality. We show that our techniques generate correct confidence
intervals on a range of real-world datasets, and demonstrate wide applicability
by using them to evict poorly performing workers, and provide confidence
intervals on the accuracy of the answers.

本文介绍了一种用于生成工人错误率估计的置信区间的技术，以更好地评估工人质量。实验结果表明该技术在真实数据集中能生成正确的置信区间，并可用于清退表现不佳的工人，并为答案的准确性提供置信区间。