Crowdsourcing systems have been used to accumulate massive amounts of labeled
data for applications such as computer vision and natural language processing.
However, because crowdsourced labeling is inherently dynamic and uncertain,
developing a technique that can work in most situations is extremely
challenging. In this paper, we introduce Crowd-Certain, a novel approach for
label aggregation in crowdsourced and ensemble learning classification tasks
that offers improved performance and computational efficiency for different
numbers of annotators and a variety of datasets. The proposed method uses the
consistency of the annotators versus a trained classifier to determine a
reliability score for each annotator. Furthermore, Crowd-Certain leverages
predicted probabilities, enabling the reuse of trained classifiers on future
sample data, thereby eliminating the need for recurrent simulation processes
inherent in existing methods. We extensively evaluated our approach against ten
existing techniques across ten different datasets, each labeled by varying
numbers of annotators. The findings demonstrate that Crowd-Certain outperforms
the existing methods (Tao, Sheng, KOS, MACE, MajorityVote, MMSR, Wawa,
Zero-Based Skill, GLAD, and Dawid Skene), in nearly all scenarios, delivering
higher average accuracy, F1 scores, and AUC rates. Additionally, we introduce a
variation of two existing confidence score measurement techniques. Finally we
evaluate these two confidence score techniques using two evaluation metrics:
Expected Calibration Error (ECE) and Brier Score Loss. Our results show that
Crowd-Certain achieves higher Brier Score, and lower ECE across the majority of
the examined datasets, suggesting better calibrated results.

Crowd-Certain 是一种在众包和集成学习分类任务中进行标签聚合的新方法，通过评估标注者的一致性和受训分类器，确定每个标注者的可靠性分数，并利用预测概率实现对未来样本数据的训练分类器的复用，大幅提升性能和计算效率。在对十个不同数据集进行了广泛评估后，Crowd-Certain 几乎在所有场景中优于现有的十种方法，提供更高的平均准确率、F1 分数和 AUC 率。另外，我们引入了两种现有置信度评估技术的变体，并通过两种评估指标（期望校准误差和 Brier 得分损失）对其进行了评估，结果表明 Crowd-Certain 在大部分数据集上具有更高的 Brier 得分和更低的校准误差，表明结果的校准更好。