XNLI is a popular Natural Language Inference (NLI) benchmark widely used to
evaluate cross-lingual Natural Language Understanding (NLU) capabilities across
languages. In this paper, we expand XNLI to include Basque, a low-resource
language that can greatly benefit from transfer-learning approaches. The new
dataset, dubbed XNLIeu, has been developed by first machine-translating the
English XNLI corpus into Basque, followed by a manual post-edition step. We
have conducted a series of experiments using mono- and multilingual LLMs to
assess a) the effect of professional post-edition on the MT system; b) the best
cross-lingual strategy for NLI in Basque; and c) whether the choice of the best
cross-lingual strategy is influenced by the fact that the dataset is built by
translation. The results show that post-edition is necessary and that the
translate-train cross-lingual strategy obtains better results overall, although
the gain is lower when tested in a dataset that has been built natively from
scratch. Our code and datasets are publicly available under open licenses.

本研究通过将英语 XNLI 语料库机器翻译成巴斯克语，然后进行人工后期处理的步骤来扩展 XNLI，从而对巴斯克语进行低资源跨语言自然语言理解进行评估，结果表明后期处理是必要的，翻译 - 训练跨语言策略在总体上获得更好的结果。

XNLIeu：巴斯克语的跨语言自然语言推理数据集

XNLIeu: a dataset for cross-lingual NLI in Basque

In the age of large transformer language models, linguistic evaluation play
an important role in diagnosing models' abilities and limitations on natural
language understanding. However, current evaluation methods show some
significant shortcomings. In particular, they do not provide insight into how
well a language model captures distinct linguistic skills essential for
language understanding and reasoning. Thus they fail to effectively map out the
aspects of language understanding that remain challenging to existing models,
which makes it hard to discover potential limitations in models and datasets.
In this paper, we introduce Curriculum as a new format of NLI benchmark for
evaluation of broad-coverage linguistic phenomena. Curriculum contains a
collection of datasets that covers 36 types of major linguistic phenomena and
an evaluation procedure for diagnosing how well a language model captures
reasoning skills for distinct types of linguistic phenomena. We show that this
linguistic-phenomena-driven benchmark can serve as an effective tool for
diagnosing model behavior and verifying model learning quality. In addition,
our experiments provide insight into the limitation of existing benchmark
datasets and state-of-the-art models that may encourage future research on
re-designing datasets, model architectures, and learning objectives.

本文介绍一种新的 NLI 基准 Curriculum，其中包括 36 种广泛涵盖的语言现象的数据集和评估程序，证明这种以语言现象驱动的基准在诊断模型行为和验证模型学习质量方面具有有效性，同时为未来对数据集的重新设计、模型架构和学习目标的研究提供了启示和借鉴。