Solving complex classification tasks using deep neural networks typically
requires large amounts of annotated data. However, corresponding class labels
are noisy when provided by error-prone annotators, e.g., crowd workers.
Training standard deep neural networks leads to subpar performances in such
multi-annotator supervised learning settings. We address this issue by
presenting a probabilistic training framework named multi-annotator deep
learning (MaDL). A ground truth and an annotator performance model are jointly
trained in an end-to-end learning approach. The ground truth model learns to
predict instances' true class labels, while the annotator performance model
infers probabilistic estimates of annotators' performances. A modular network
architecture enables us to make varying assumptions regarding annotators'
performances, e.g., an optional class or instance dependency. Further, we learn
annotator embeddings to estimate annotators' densities within a latent space as
proxies of their potentially correlated annotations. Together with a weighted
loss function, we improve the learning from correlated annotation patterns. In
a comprehensive evaluation, we examine three research questions about
multi-annotator supervised learning. Our findings indicate MaDL's
state-of-the-art performance and robustness against many correlated, spamming
annotators.

我们提出了一个名为 MaDL 的概率训练框架，其中联合训练了一个基础真实模型和一个注释者模型来解决多注释者监督学习中存在的嘈杂标签问题，并获得了最先进的性能。