Disagreement in natural language annotation has mostly been studied from a
perspective of biases introduced by the annotators and the annotation
frameworks. Here, we propose to analyze another source of bias: task design
bias, which has a particularly strong impact on crowdsourced linguistic
annotations where natural language is used to elicit the interpretation of
laymen annotators. For this purpose we look at implicit discourse relation
annotation, a task that has repeatedly been shown to be difficult due to the
relations' ambiguity. We compare the annotations of 1,200 discourse relations
obtained using two distinct annotation tasks and quantify the biases of both
methods across four different domains. Both methods are natural language
annotation tasks designed for crowdsourcing. We show that the task design can
push annotators towards certain relations and that some discourse relations
senses can be better elicited with one or the other annotation approach. We
also conclude that this type of bias should be taken into account when training
and testing models.

本文分析了任务设计偏差对自然语言注释的影响，结论表明任务设计对注释人员对关系的选择有影响，且应将这种偏差考虑到模型的训练和测试中。