Language models (LMs) have introduced a major paradigm shift in Natural
Language Processing (NLP) modeling where large pre-trained LMs became integral
to most of the NLP tasks. The LMs are intelligent enough to find useful and
relevant representations of the language without any supervision. Perhaps,
these models are used to fine-tune typical NLP tasks with significantly high
accuracy as compared to the traditional approaches. Conversely, the training of
these models requires a massively large corpus that is a good representation of
the language. English LMs generally perform better than their other language
counterparts, due to the availability of massive English corpora. This work
elaborates on the design and development of a large Arabic corpus. It consists
of over 500 GB of Arabic cleaned text targeted at improving cross-domain
knowledge and downstream generalization capability of large-scale language
models. Moreover, the corpus is utilized in the training of a large Arabic LM.
In order to evaluate the effectiveness of the LM, a number of typical NLP tasks
are fine-tuned. The tasks demonstrate a significant boost from 4.5 to 8.5% when
compared to tasks fine-tuned on multi-lingual BERT (mBERT). To the best of my
knowledge, this is currently the largest clean and diverse Arabic corpus ever
collected.

这项研究介绍了一个由超过 500GB 的阿拉伯语言清理文本构成的语料库，旨在提高大规模语言模型的跨领域知识和下游泛化能力。此外，该语料库还被用于大型阿拉伯语言模型的训练，在对典型的 NLP 任务进行微调时，与 mBERT 相比表现出 4.5% 至 8.5% 的显着提升，据我所知，这是目前所收集的最大、最清洁、最具多样性的阿拉伯语语料库。

面向语言建模的大规模多样化阿拉伯语语料库

A Large and Diverse Arabic Corpus for Language Modeling

This work explores the application of textual entailment in news claim
verification and stance prediction using a new corpus in Arabic. The publicly
available corpus comes in two perspectives: a version consisting of 4,547 true
and false claims and a version consisting of 3,786 pairs (claim, evidence). We
describe the methodology for creating the corpus and the annotation process.
Using the introduced corpus, we also develop two machine learning baselines for
two proposed tasks: claim verification and stance prediction. Our best model
utilizes pretraining (BERT) and achieves 76.7 F1 on the stance prediction task
and 64.3 F1 on the claim verification task. Our preliminary experiments shed
some light on the limits of automatic claim verification that relies on claims
text only. Results hint that while the linguistic features and world knowledge
learned during pretraining are useful for stance prediction, such learned
representations from pretraining are insufficient for verifying claims without
access to context or evidence.

该论文探讨了在阿拉伯语新闻断言验证和观点预测中使用文本蕴含的应用，并使用一个新的语料库。作者介绍了语料库的创建方法和注释过程，开发了用于两个提出的任务的两个机器学习基线：断言验证和立场预测。作者的最佳模型利用预训练（BERT），在立场预测任务上实现了 76.7 F1，在断言验证任务上实现了 64.3 F1。结果表明，尽管通过预训练学习到的语言特征和世界知识对于立场预测是有用的，但是这些学习表示对于没有上下文或证据的断言验证是不够的。

立场预测和主张验证：阿拉伯视角

Stance Prediction and Claim Verification: An Arabic Perspective

This study is an attempt to build a contemporary linguistic corpus for Arabic
language. The corpus produced, is a text corpus includes more than five million
newspaper articles. It contains over a billion and a half words in total, out
of which, there is about three million unique words. The data were collected
from newspaper articles in ten major news sources from eight Arabic countries,
over a period of fourteen years. The corpus was encoded with two types of
encoding, namely: UTF-8, and Windows CP-1256. Also it was marked with two
mark-up languages, namely: SGML, and XML.

本研究旨在建立一份当代阿拉伯文语料库，收集了包括 8 个阿拉伯国家 10 个主要新闻来源的报纸文章，共计超过 5 百万篇文章，包含 15 亿个单词，标记了 SGML 和 XML 两种标记语言，并使用 UTF-8 和 Windows CP-1256 两种编码格式。