Pretraining monolingual language models have been proven to be vital for
performance in Arabic Natural Language Processing (NLP) tasks. In this paper,
we conduct a comprehensive study on the role of data in Arabic Pretrained
Language Models (PLMs). More precisely, we reassess the performance of a suite
of state-of-the-art Arabic PLMs by retraining them on massive-scale,
high-quality Arabic corpora. We have significantly improved the performance of
the leading Arabic encoder-only BERT-base and encoder-decoder T5-base models on
the ALUE and ORCA leaderboards, thereby reporting state-of-the-art results in
their respective model categories. In addition, our analysis strongly suggests
that pretraining data by far is the primary contributor to performance,
surpassing other factors. Our models and source code are publicly available at
this https URL

通过在规模庞大、高质量的阿拉伯语语料库上重新训练，我们显著提高了领先的阿拉伯编码器型 BERT-based 和编码器解码器型 T5-based 的性能，表现出同类模型的最新技术结果；我们的分析强烈暗示，预训练数据远远是性能的主要贡献因素，超越其他因素。