Over the last few years, deep learning has revolutionized the field of
machine learning by dramatically improving the state-of-the-art in various
domains. However, as the size of supervised artificial neural networks grows,
typically so does the need for larger labeled datasets. Recently, crowdsourcing
has established itself as an efficient and cost-effective solution for labeling
large sets of data in a scalable manner, but it often requires aggregating
labels from multiple noisy contributors with different levels of expertise. In
this paper, we address the problem of learning deep neural networks from
crowds. We begin by describing an EM algorithm for jointly learning the
parameters of the network and the reliabilities of the annotators. Then, a
novel general-purpose crowd layer is proposed, which allows us to train deep
neural networks end-to-end, directly from the noisy labels of multiple
annotators, using only backpropagation. We empirically show that the proposed
approach is able to internally capture the reliability and biases of different
annotators and achieve new state-of-the-art results for various crowdsourced
datasets across different settings, namely classification, regression and
sequence labeling.

本文提出了一种基于 EM 算法和众包技术的神经网络训练模型，能够直接从多个标注者的嘈杂标签数据中进行有监督学习，并能够捕捉不同标注者的可靠性和偏见，最终在多个领域获得了新的最优结果。

众包深度学习

Deep learning from crowds

Manual annotations of temporal bounds for object interactions (i.e. start and
end times) are typical training input to recognition, localization and
detection algorithms. For three publicly available egocentric datasets, we
uncover inconsistencies in ground truth temporal bounds within and across
annotators and datasets. We systematically assess the robustness of
state-of-the-art approaches to changes in labeled temporal bounds, for object
interaction recognition. As boundaries are trespassed, a drop of up to 10% is
observed for both Improved Dense Trajectories and Two-Stream Convolutional
Neural Network.
We demonstrate that such disagreement stems from a limited understanding of
the distinct phases of an action, and propose annotating based on the Rubicon
Boundaries, inspired by a similarly named cognitive model, for consistent
temporal bounds of object interactions. Evaluated on a public dataset, we
report a 4% increase in overall accuracy, and an increase in accuracy for 55%
of classes when Rubicon Boundaries are used for temporal annotations.

通过研究三个虚拟数据集的标注者在标注时间边界上的一致性问题，我们提出了一种使用基于认知模型的 “Rubicon Boundaries” 来更加一致地标注时间边界的方法，并在一个公共数据集上取得了总准确度提升 4% 和 55% 类别准确度提升的效果。

越界：自我中心视角视频中对象交互的时间边界标注

Trespassing the Boundaries: Labeling Temporal Bounds for Object  Interactions in Egocentric Video

With the rapid growth of crowdsourcing platforms it has become easy and
relatively inexpensive to collect a dataset labeled by multiple annotators in a
short time. However due to the lack of control over the quality of the
annotators, some abnormal annotators may be affected by position bias which can
potentially degrade the quality of the final consensus labels. In this paper we
introduce a statistical framework to model and detect annotator's position bias
in order to control the false discovery rate (FDR) without a prior knowledge on
the amount of biased annotators - the expected fraction of false discoveries
among all discoveries being not too high, in order to assure that most of the
discoveries are indeed true and replicable. The key technical development
relies on some new knockoff filters adapted to our problem and new algorithms
based on the Inverse Scale Space dynamics whose discretization is potentially
suitable for large scale crowdsourcing data analysis. Our studies are supported
by experiments with both simulated examples and real-world data. The proposed
framework provides us a useful tool for quantitatively studying annotator's
abnormal behavior in crowdsourcing data arising from machine learning,
sociology, computer vision, multimedia, etc.

本文中，我们开发了一个统计框架来建模和检测标注者的位置偏差，并采用新型 knockoff 过滤器和基于反比例尺度动力学的算法来控制虚警率，为定量研究众包数据中的标注者异常行为提供了一种有用的工具。

众包排名中的假发现率控制与标注者统计质量评估

False Discovery Rate Control and Statistical Quality Assessment of  Annotators in Crowdsourced Ranking

To learn semantic attributes, existing methods typically train one
discriminative model for each word in a vocabulary of nameable properties.
However, this "one model per word" assumption is problematic: while a word
might have a precise linguistic definition, it need not have a precise visual
definition. We propose to discover shades of attribute meaning. Given an
attribute name, we use crowdsourced image labels to discover the latent factors
underlying how different annotators perceive the named concept. We show that
structure in those latent factors helps reveal shades, that is, interpretations
for the attribute shared by some group of annotators. Using these shades, we
train classifiers to capture the primary (often subtle) variants of the
attribute. The resulting models are both semantic and visually precise. By
catering to users' interpretations, they improve attribute prediction accuracy
on novel images. Shades also enable more successful attribute-based image
search, by providing robust personalized models for retrieving multi-attribute
query results. They are widely applicable to tasks that involve describing
visual content, such as zero-shot category learning and organization of photo
collections.

该研究提出一种使用众包图像标签发现属性名称的潜在因素的方法，以捕捉属性的主要变体，从而提高基于属性的图像搜索的精度，以及用于描述视觉内容的一般任务中的广泛应用。