Crowdsourcing platforms use various truth discovery algorithms to aggregate
annotations from multiple labelers. In an online setting, however, the main
challenge is to decide whether to ask for more annotations for each item to
efficiently trade off cost (i.e., the number of annotations) for quality of the
aggregated annotations. In this paper, we propose a novel approach for general
complex annotation (such as bounding boxes and taxonomy paths), that works in
an online crowdsourcing setting. We prove that the expected average similarity
of a labeler is linear in their accuracy \emph{conditional on the reported
label}. This enables us to infer reported label accuracy in a broad range of
scenarios. We conduct extensive evaluations on real-world crowdsourcing data
from Meta and show the effectiveness of our proposed online algorithms in
improving the cost-quality trade-off.

在线众包平台的真实性发现算法在聚合多个标注人的注释时起着重要作用，本论文提出了一种适用于在线众包环境的新方法，能够在有效平衡成本（即标注数量）和注释质量的同时，对一般复杂注释（如边界框和分类路径）进行推断，该方法以 label 者的准确度为线性期望平均相似度，经过对 Meta 真实众包数据的广泛评估后，展示了我们提出的在线算法在改善成本 - 质量权衡方面的有效性。

高效在线众包任务与复杂注释

Efficient Online Crowdsourcing with Complex Annotations

Majority voting is a simple mathematical function that returns the value that
appears most often in a set. As a popular decision fusion technique, the
majority voting function (MVF) finds applications in resolving conflicts, where
a number of independent voters report their opinions on a classification
problem. Despite its importance and its various applications in ensemble
learning, data crowd-sourcing, remote sensing, and data oracles for
blockchains, the accuracy of the MVF for the general multi-class classification
problem has remained unknown. In this paper, we derive a new upper bound on the
accuracy of the MVF for the multi-class classification problem. More
specifically, we show that under certain conditions, the error rate of the MVF
exponentially decays toward zero as the number of independent voters increases.
Conversely, the error rate of the MVF exponentially grows with the number of
independent voters if these conditions are not met.
We first explore the problem for independent and identically distributed
voters where we assume that every voter follows the same conditional
probability distribution of voting for different classes, given the true
classification of the data point. Next, we extend our results for the case
where the voters are independent but non-identically distributed. Using the
derived results, we then provide a discussion on the accuracy of the truth
discovery algorithms. We show that in the best-case scenarios, truth discovery
algorithms operate as an amplified MVF and thereby achieve a small error rate
only when the MVF achieves a small error rate, and vice versa, achieve a large
error rate when the MVF also achieves a large error rate. In the worst-case
scenario, the truth discovery algorithms may achieve a higher error rate than
the MVF. Finally, we confirm our theoretical results using numerical
simulations.

绝对多数投票函数（Majority Voting Function）在多类别分类问题中的精度存在上界，并且在某些条件下随着独立投票者的数量增加，错误率指数级地趋近于零，否则错误率也指数级地增加，同时提到了整合算法的精度和真实发现算法的问题。