Cognitive psychologists have documented that humans use cognitive heuristics, or mental shortcuts, to make quick decisions while expending less effort. While performing annotation work on crowdsourcing platforms, we hypothesize that such heuristic use among annotators cascades on to data quality and model robustness. In this work, we study cognitive heuristic use in the context of annotating multiple-choice reading comprehension datasets. We propose tracking annotator heuristic traces, where we tangibly measure low-effort annotation strategies that could indicate usage of various cognitive heuristics. We find evidence that annotators might be using multiple such heuristics, based on correlations with a battery of psychological tests. Importantly, heuristic use among annotators determines data quality along several dimensions: (1) known biased models, such as partial input models, more easily solve examples authored by annotators that rate highly on heuristic use, (2) models trained on annotators scoring highly on heuristic use don't generalize as well, and (3) heuristic-seeking annotators tend to create qualitatively less challenging examples. Our findings suggest that tracking heuristic usage among annotators can potentially help with collecting challenging datasets and diagnosing model biases.

本文探讨了在众包平台上进行注释任务时，注释者使用认知启发式对数据质量和模型稳健性的影响，并提出了追踪注释者启发式痕迹以确定使用各种认知启发式的方法。作者发现，注释者可能会使用多种认知启发式，由其决定数据质量并影响模型的鲁棒性。追踪注释者使用认知启发式有可能帮助收集有挑战性的数据集并诊断模型偏差。

瀑布偏差：探讨启发式标注策略对数据和模型的影响