Current practices regarding data collection for natural language processing
on Amazon Mechanical Turk (MTurk) often rely on a combination of studies on
data quality and heuristics shared among NLP researchers. However, without
considering the perspectives of MTurk workers, these approaches are susceptible
to issues regarding workers' rights and poor response quality. We conducted a
critical literature review and a survey of MTurk workers aimed at addressing
open questions regarding best practices for fair payment, worker privacy, data
quality, and considering worker incentives. We found that worker preferences
are often at odds with received wisdom among NLP researchers. Surveyed workers
preferred reliable, reasonable payments over uncertain, very high payments;
reported frequently lying on demographic questions; and expressed frustration
at having work rejected with no explanation. We also found that workers view
some quality control methods, such as requiring minimum response times or
Master's qualifications, as biased and largely ineffective. Based on the survey
results, we provide recommendations on how future NLP studies may better
account for MTurk workers' experiences in order to respect workers' rights and
improve data quality.

通过批判性文献综述和 MTurk 工作者的调查，我们发现当前在亚马逊众包平台中进行自然语言处理数据收集的方法在考虑工作者的角度上存在问题，因此我们提出了关于公平支付、工作者隐私、数据质量和工作者激励等方面的最佳实践，以更好地尊重工作者权益并提高数据质量。