Pre-trained language models (LMs) are currently integral to many natural
language processing systems. Although multilingual LMs were also introduced to
serve many languages, these have limitations such as being costly at inference
time and the size and diversity of non-English data involved in their
pre-training. We remedy these issues for a collection of diverse Arabic
varieties by introducing two powerful deep bidirectional transformer-based
models, ARBERT and MARBERT. To evaluate our models, we also introduce ARLUE, a
new benchmark for multi-dialectal Arabic language understanding evaluation.
ARLUE is built using 42 datasets targeting six different task clusters,
allowing us to offer a series of standardized experiments under rich
conditions. When fine-tuned on ARLUE, our models collectively achieve new
state-of-the-art results across the majority of tasks (37 out of 48
classification tasks, on the 42 datasets). Our best model acquires the highest
ARLUE score (77.40) across all six task clusters, outperforming all other
models including XLM-R Large (~ 3.4 x larger size). Our models are publicly
available at this https URL and ARLUE will be released
through the same repository.

本文提出了两个基于深度双向变压器的模型 ARBERT 和 MARBERT，并引入了一个针对多语言阿拉伯语理解评估的基准 ARLUE。在 42 个数据集上进行了一系列基准实验，结果表明将这两种模型细化后在大多数任务上取得了最先进的结果并提供了公共可用的模型。