Many crowdsourced NLP datasets contain systematic gaps and biases that are identified only after data collection is complete. Identifying these issues from early data samples during crowdsourcing should make mitigation more efficient, especially when done iteratively. We take natural language inference as a test case and ask whether it is beneficial to put a linguist `in the loop' during data collection to dynamically identify and address gaps in the data by introducing novel constraints on the task. We directly compare three data collection protocols: (i) a baseline protocol, (ii) a linguist-in-the-loop intervention with iteratively-updated constraints on the task, and (iii) an extension of linguist-in-the-loop that provides direct interaction between linguists and crowdworkers via a chatroom. The datasets collected with linguist involvement are more reliably challenging than baseline, without loss of quality. But we see no evidence that using this data in training leads to better out-of-domain model performance, and the addition of a chat platform has no measurable effect on the resulting dataset. We suggest integrating expert analysis \textit{during} data collection so that the expert can dynamically address gaps and biases in the dataset.

本研究以自然语言推理为测试案例，探究在众包数据收集过程中加入语言学家以动态地识别和解决数据中的漏洞和偏见是否有利于减轻偏见，提高数据的可靠性。对比了三种数据收集方案，发现加入语言学家对数据的挑战性的提高不会将数据的质量降低，并提出建议在数据收集期间整合专家的分析以解决数据中的偏见和系统漏洞。

将语言学家参与是否提高 NLU 数据收集的效果？