We present the HPLT (High Performance Language Technologies) language
resources, a new massive multilingual dataset including both monolingual and
bilingual corpora extracted from CommonCrawl and previously unused web crawls
from the Internet Archive. We describe our methods for data acquisition,
management and processing of large corpora, which rely on open-source software
tools and high-performance computing. Our monolingual collection focuses on
low- to medium-resourced languages and covers 75 languages and a total of ~5.6
trillion word tokens de-duplicated on the document level. Our English-centric
parallel corpus is derived from its monolingual counterpart and covers 18
language pairs and more than 96 million aligned sentence pairs with roughly 1.4
billion English tokens. The HPLT language resources are one of the largest open
text corpora ever released, providing a great resource for language modeling
and machine translation training. We publicly release the corpora, the
software, and the tools used in this work.

我们介绍了 HPLT（高性能语言技术）语言资源，这是一个包括来自 CommonCrawl 和互联网档案的先前未使用的网络爬取的单语和双语语料库的大规模多语言数据集。我们描述了我们采集、管理和处理大型语料库的方法，这些方法依赖于开源软件工具和高性能计算。我们的单语语料库主要关注资源较低到中等的语言，涵盖 75 种语言和总共大约 5.6 万亿个单词（根据文档级别去重）。我们的以英语为中心的平行语料库源自于单语语料库，涵盖 18 种语言对，包括超过 9600 万个对齐的句子对和大约 14 亿个英语单词。HPLT 语言资源是迄今为止史上最大的开放文本语料库之一，为语言建模和机器翻译训练提供了极好的资源。我们公开发布了本工作中使用的语料库、软件和工具。