We present IndoNLI, the first human-elicited NLI dataset for Indonesian. We
adapt the data collection protocol for MNLI and collect nearly 18K sentence
pairs annotated by crowd workers and experts. The expert-annotated data is used
exclusively as a test set. It is designed to provide a challenging test-bed for
Indonesian NLI by explicitly incorporating various linguistic phenomena such as
numerical reasoning, structural changes, idioms, or temporal and spatial
reasoning. Experiment results show that XLM-R outperforms other pre-trained
models in our data. The best performance on the expert-annotated data is still
far below human performance (13.4% accuracy gap), suggesting that this test set
is especially challenging. Furthermore, our analysis shows that our
expert-annotated data is more diverse and contains fewer annotation artifacts
than the crowd-annotated data. We hope this dataset can help accelerate
progress in Indonesian NLP research.

我们提出了 IndoNLI 数据集，这是第一个人类收集的适用于印度尼西亚语的 NLI 数据集。我们采用 MNLI 的数据收集协议，收集了近 18K 个由众包工人和专家标注的句子对。实验结果表明，在我们的数据中，XLM-R 的性能优于其他预训练模型。最佳性能仍然远低于人类性能（13.4％的准确度差距），表明此测试集具有特别的挑战性。此外，我们的分析显示，我们的专家标注数据比众包标注数据更加多样化，注释工件更少。我们希望这个数据集可以帮助加快印尼自然语言处理研究的进展。