While many natural language inference (NLI) datasets target certain semantic
phenomena, e.g., negation, tense & aspect, monotonicity, and presupposition, to
the best of our knowledge, there is no NLI dataset that involves diverse types
of spatial expressions and reasoning. We fill this gap by semi-automatically
creating an NLI dataset for spatial reasoning, called SpaceNLI. The data
samples are automatically generated from a curated set of reasoning patterns,
where the patterns are annotated with inference labels by experts. We test
several SOTA NLI systems on SpaceNLI to gauge the complexity of the dataset and
the system's capacity for spatial reasoning. Moreover, we introduce a Pattern
Accuracy and argue that it is a more reliable and stricter measure than the
accuracy for evaluating a system's performance on pattern-based generated data
samples. Based on the evaluation results we find that the systems obtain
moderate results on the spatial NLI problems but lack consistency per inference
pattern. The results also reveal that non-projective spatial inferences
(especially due to the "between" preposition) are the most challenging ones.

通过语义推理研究各种类型的空间表达方式与推理模式，我们创建了一个名为 SpaceNLI 的数据集，并测试了几种 NLI 系统的它们对于空间推理复杂度和系统能力的提高。通过评估结果，我们发现系统对于空间 NLI 问题获得了适度的结果，但缺乏推理模式的一致性。结果还表明，非射影空间推理（尤其是由 “between” 介词引起的）是最具挑战性的。

SpaceNLI: 评估推理预测的一致性

SpaceNLI: Evaluating the Consistency of Predicting Inferences in Space

While Indic NLP has made rapid advances recently in terms of the availability
of corpora and pre-trained models, benchmark datasets on standard NLU tasks are
limited. To this end, we introduce IndicXNLI, an NLI dataset for 11 Indic
languages. It has been created by high-quality machine translation of the
original English XNLI dataset and our analysis attests to the quality of
IndicXNLI. By finetuning different pre-trained LMs on this IndicXNLI, we
analyze various cross-lingual transfer techniques with respect to the impact of
the choice of language models, languages, multi-linguality, mix-language input,
etc. These experiments provide us with useful insights into the behaviour of
pre-trained models for a diverse set of languages.

本文介绍了 IndicXNLI，一个 NLI 数据集，用于 11 种印度语言的跨语言转移技术的分析，研究了不同的预训练模型、语言、多语言和混合语言输入等因素对预训练模型的行为的影响。

IndicXNLI: 评估印度语言的多语言推理

IndicXNLI: Evaluating Multilingual Inference for Indian Languages

We present IndoNLI, the first human-elicited NLI dataset for Indonesian. We
adapt the data collection protocol for MNLI and collect nearly 18K sentence
pairs annotated by crowd workers and experts. The expert-annotated data is used
exclusively as a test set. It is designed to provide a challenging test-bed for
Indonesian NLI by explicitly incorporating various linguistic phenomena such as
numerical reasoning, structural changes, idioms, or temporal and spatial
reasoning. Experiment results show that XLM-R outperforms other pre-trained
models in our data. The best performance on the expert-annotated data is still
far below human performance (13.4% accuracy gap), suggesting that this test set
is especially challenging. Furthermore, our analysis shows that our
expert-annotated data is more diverse and contains fewer annotation artifacts
than the crowd-annotated data. We hope this dataset can help accelerate
progress in Indonesian NLP research.

我们提出了 IndoNLI 数据集，这是第一个人类收集的适用于印度尼西亚语的 NLI 数据集。我们采用 MNLI 的数据收集协议，收集了近 18K 个由众包工人和专家标注的句子对。实验结果表明，在我们的数据中，XLM-R 的性能优于其他预训练模型。最佳性能仍然远低于人类性能（13.4％的准确度差距），表明此测试集具有特别的挑战性。此外，我们的分析显示，我们的专家标注数据比众包标注数据更加多样化，注释工件更少。我们希望这个数据集可以帮助加快印尼自然语言处理研究的进展。