Due to their crucial role in all NLP, several benchmarks have been proposed
to evaluate pretrained language models. In spite of these efforts, no public
benchmark of diverse nature currently exists for evaluation of Arabic. This
makes it challenging to measure progress for both Arabic and multilingual
language models. This challenge is compounded by the fact that any benchmark
targeting Arabic needs to take into account the fact that Arabic is not a
single language but rather a collection of languages and varieties. In this
work, we introduce ORCA, a publicly available benchmark for Arabic language
understanding evaluation. ORCA is carefully constructed to cover diverse Arabic
varieties and a wide range of challenging Arabic understanding tasks exploiting
60 different datasets across seven NLU task clusters. To measure current
progress in Arabic NLU, we use ORCA to offer a comprehensive comparison between
18 multilingual and Arabic language models. We also provide a public
leaderboard with a unified single-number evaluation metric (ORCA score) to
facilitate future research.

介绍了一个公开的基准测试 ORCA 用于评估阿拉伯语言理解，利用 60 个数据集跨越 7 个 NLU 任务簇，用 ORCA 评估了 18 个多语言和阿拉伯语言模型，提供了一个统一的单一数字评估指标。

ORCA：阿拉伯语言理解挑战基准

ORCA: A Challenging Benchmark for Arabic Language Understanding

Pre-trained language models (LMs) are currently integral to many natural
language processing systems. Although multilingual LMs were also introduced to
serve many languages, these have limitations such as being costly at inference
time and the size and diversity of non-English data involved in their
pre-training. We remedy these issues for a collection of diverse Arabic
varieties by introducing two powerful deep bidirectional transformer-based
models, ARBERT and MARBERT. To evaluate our models, we also introduce ARLUE, a
new benchmark for multi-dialectal Arabic language understanding evaluation.
ARLUE is built using 42 datasets targeting six different task clusters,
allowing us to offer a series of standardized experiments under rich
conditions. When fine-tuned on ARLUE, our models collectively achieve new
state-of-the-art results across the majority of tasks (37 out of 48
classification tasks, on the 42 datasets). Our best model acquires the highest
ARLUE score (77.40) across all six task clusters, outperforming all other
models including XLM-R Large (~ 3.4 x larger size). Our models are publicly
available at this https URL and ARLUE will be released
through the same repository.

本文提出了两个基于深度双向变压器的模型 ARBERT 和 MARBERT，并引入了一个针对多语言阿拉伯语理解评估的基准 ARLUE。在 42 个数据集上进行了一系列基准实验，结果表明将这两种模型细化后在大多数任务上取得了最先进的结果并提供了公共可用的模型。