Annotation through crowdsourcing draws incremental attention, which relies on
an effective selection scheme given a pool of workers. Existing methods propose
to select workers based on their performance on tasks with ground truth, while
two important points are missed. 1) The historical performances of workers in
other tasks. In real-world scenarios, workers need to solve a new task whose
correlation with previous tasks is not well-known before the training, which is
called cross-domain. 2) The dynamic worker performance as workers will learn
from the ground truth. In this paper, we consider both factors in designing an
allocation scheme named cross-domain-aware worker selection with training
approach. Our approach proposes two estimation modules to both statistically
analyze the cross-domain correlation and simulate the learning gain of workers
dynamically. A framework with a theoretical analysis of the worker elimination
process is given. To validate the effectiveness of our methods, we collect two
novel real-world datasets and generate synthetic datasets. The experiment
results show that our method outperforms the baselines on both real-world and
synthetic datasets.

本文介绍了一种名为 “跨领域感知的工人选择与训练方法” 的分配方案，该方案考虑了工人在其他任务中的历史表现以及工人在任务中通过学习获得的动态表现，并通过两个评估模块统计分析了跨领域相关性和工人的学习收益。实验证明，该方法在真实和合成数据集上均超越了基准方法。