Language models (LMs) have introduced a major paradigm shift in Natural
Language Processing (NLP) modeling where large pre-trained LMs became integral
to most of the NLP tasks. The LMs are intelligent enough to find useful and
relevant representations of the language without any supervision. Perhaps,
these models are used to fine-tune typical NLP tasks with significantly high
accuracy as compared to the traditional approaches. Conversely, the training of
these models requires a massively large corpus that is a good representation of
the language. English LMs generally perform better than their other language
counterparts, due to the availability of massive English corpora. This work
elaborates on the design and development of a large Arabic corpus. It consists
of over 500 GB of Arabic cleaned text targeted at improving cross-domain
knowledge and downstream generalization capability of large-scale language
models. Moreover, the corpus is utilized in the training of a large Arabic LM.
In order to evaluate the effectiveness of the LM, a number of typical NLP tasks
are fine-tuned. The tasks demonstrate a significant boost from 4.5 to 8.5% when
compared to tasks fine-tuned on multi-lingual BERT (mBERT). To the best of my
knowledge, this is currently the largest clean and diverse Arabic corpus ever
collected.

这项研究介绍了一个由超过 500GB 的阿拉伯语言清理文本构成的语料库，旨在提高大规模语言模型的跨领域知识和下游泛化能力。此外，该语料库还被用于大型阿拉伯语言模型的训练，在对典型的 NLP 任务进行微调时，与 mBERT 相比表现出 4.5% 至 8.5% 的显着提升，据我所知，这是目前所收集的最大、最清洁、最具多样性的阿拉伯语语料库。