Natural language inference (NLI) data has proven useful in benchmarking and,
especially, as pretraining data for tasks requiring language understanding.
However, the crowdsourcing protocol that was used to collect this data has
known issues and was not explicitly optimized for either of these purposes, so
it is likely far from ideal. We propose four alternative protocols, each aimed
at improving either the ease with which annotators can produce sound training
examples or the quality and diversity of those examples. Using these
alternatives and a fifth baseline protocol, we collect and compare five new
8.5k-example training sets. In evaluations focused on transfer learning
applications, our results are solidly negative, with models trained on our
baseline dataset yielding good transfer performance to downstream tasks, but
none of our four new methods (nor the recent ANLI) showing any improvements
over that baseline. In a small silver lining, we observe that all four new
protocols, especially those where annotators edit pre-filled text boxes, reduce
previously observed issues with annotation artifacts.

该研究提出了四种新的众包协议来收集 NLI 数据集，以改善训练样本的质量和多样性，然而实验结果表明这些新方法在迁移学习的任务中没有优势，但它们可以显著降低注释文本中的问题。