Supervised classification algorithms are used to solve a growing number of
real-life problems around the globe. Their performance is strictly connected
with the quality of labels used in training. Unfortunately, acquiring
good-quality annotations for many tasks is infeasible or too expensive to be
done in practice. To tackle this challenge, active learning algorithms are
commonly employed to select only the most relevant data for labeling. However,
this is possible only when the quality and quantity of labels acquired from
experts are sufficient. Unfortunately, in many applications, a trade-off
between annotating individual samples by multiple annotators to increase label
quality vs. annotating new samples to increase the total number of labeled
instances is necessary. In this paper, we address the issue of faulty data
annotations in the context of active learning. In particular, we propose two
novel annotation unification algorithms that utilize unlabeled parts of the
sample space. The proposed methods require little to no intersection between
samples annotated by different experts. Our experiments on four public datasets
indicate the robustness and superiority of the proposed methods in both, the
estimation of the annotator's reliability, and the assignment of actual labels,
against the state-of-the-art algorithms and the simple majority voting.

利用未标记的样本空间，我们提出了两种新的注释统一算法，旨在解决主动学习中存在的错误数据标注问题，并通过在四个公共数据集上的实验证明该方法在评估标注者可靠性和分配实际标签方面的鲁棒性和优越性，超过了现有算法和简单的多数投票。