In the recent years, transformer-based models have lead to significant
advances in language modelling for natural language processing. However, they
require a vast amount of data to be (pre-)trained and there is a lack of
corpora in languages other than English. Recently, several initiatives have
presented multilingual datasets obtained from automatic web crawling. However,
the results in Spanish present important shortcomings, as they are either too
small in comparison with other languages, or present a low quality derived from
sub-optimal cleaning and deduplication. In this paper, we introduce esCorpius,
a Spanish crawling corpus obtained from near 1 Pb of Common Crawl data. It is
the most extensive corpus in Spanish with this level of quality in the
extraction, purification and deduplication of web textual content. Our data
curation process involves a novel highly parallel cleaning pipeline and
encompasses a series of deduplication mechanisms that together ensure the
integrity of both document and paragraph boundaries. Additionally, we maintain
both the source web page URL and the WARC shard origin URL in order to complain
with EU regulations. esCorpius has been released under CC BY-NC-ND 4.0 license
and is available on HuggingFace.

通过使用多达 1 PB 的 Common Crawl 数据，我们提出了 esCorpius，一个高质量的西班牙网络爬取语料库，利用了新颖的高度并行的清理管道和多种去重机制，以确保文档和段落边界的完整性，并遵守欧盟法规。