Whose labels should a machine learning (ML) algorithm learn to emulate? For
ML tasks ranging from online comment toxicity to misinformation detection to
medical diagnosis, different groups in society may have irreconcilable
disagreements about ground truth labels. Supervised ML today resolves these
label disagreements implicitly using majority vote, which overrides minority
groups' labels. We introduce jury learning, a supervised ML approach that
resolves these disagreements explicitly through the metaphor of a jury:
defining which people or groups, in what proportion, determine the classifier's
prediction. For example, a jury learning model for online toxicity might
centrally feature women and Black jurors, who are commonly targets of online
harassment. To enable jury learning, we contribute a deep learning architecture
that models every annotator in a dataset, samples from annotators' models to
populate the jury, then runs inference to classify. Our architecture enables
juries that dynamically adapt their composition, explore counterfactuals, and
visualize dissent.

本文提出通过采用陪审团的隐喻来确立不同的人群在机器学习算法中的权威性，设计了一个可以模拟数据标记者的深度学习结构，在解决标签不一致性的过程中具有更灵活的适应性和探索性。

陪审学习：将异议声纳入机器学习模型中

Jury Learning: Integrating Dissenting Voices into Machine Learning Models

The predictive performance of supervised learning algorithms depends on the
quality of labels. In a typical label collection process, multiple annotators
provide subjective noisy estimates of the "truth" under the influence of their
varying skill-levels and biases. Blindly treating these noisy labels as the
ground truth limits the accuracy of learning algorithms in the presence of
strong disagreement. This problem is critical for applications in domains such
as medical imaging where both the annotation cost and inter-observer
variability are high. In this work, we present a method for simultaneously
learning the individual annotator model and the underlying true label
distribution, using only noisy observations. Each annotator is modeled by a
confusion matrix that is jointly estimated along with the classifier
predictions. We propose to add a regularization term to the loss function that
encourages convergence to the true annotator confusion matrix. We provide a
theoretical argument as to how the regularization is essential to our approach
both for the case of single annotator and multiple annotators. Despite the
simplicity of the idea, experiments on image classification tasks with both
simulated and real labels show that our method either outperforms or performs
on par with the state-of-the-art methods and is capable of estimating the
skills of annotators even with a single label available per image.

本文研究了如何解决多个注释器提供的有噪声的标签的问题，提出了一种同时学习个体注释器模型和真实标签分布的方法，并通过正则化项促进收敛于真实注释器混淆矩阵的方法，在图像分类任务上实验结果表明，该方法能够估计注释器的技能并表现出良好的性能。