Large-scale natural language inference (NLI) datasets such as SNLI or MNLI
have been created by asking crowdworkers to read a premise and write three new
hypotheses, one for each possible semantic relationships (entailment,
contradiction, and neutral). While this protocol has been used to create useful
benchmark data, it remains unclear whether the writing-based annotation
protocol is optimal for any purpose, since it has not been evaluated directly.
Furthermore, there is ample evidence that crowdworker writing can introduce
artifacts in the data. We investigate two alternative protocols which
automatically create candidate (premise, hypothesis) pairs for annotators to
label. Using these protocols and a writing-based baseline, we collect several
new English NLI datasets of over 3k examples each, each using a fixed amount of
annotator time, but a varying number of examples to fit that time budget. Our
experiments on NLI and transfer learning show negative results: None of the
alternative protocols outperforms the baseline in evaluations of generalization
within NLI or on transfer to outside target tasks. We conclude that crowdworker
writing still the best known option for entailment data, highlighting the need
for further data collection work to focus on improving writing-based annotation
processes.

本研究探讨了三种自然语言推理数据集的构建方法 ——crowdworker writing，自动构建以及折衷方案，并在此基础上收集了新的英文 NLI 数据集。实验结果表明，crowdworker writing 仍是 NLI 数据集构建的最佳选择。