Cross-lingual document alignment aims to identify pairs of documents in two
distinct languages that are of comparable content or translations of each
other. In this paper, we exploit the signals embedded in URLs to label web
documents at scale with an average precision of 94.5% across different language
pairs. We mine sixty-eight snapshots of the Common Crawl corpus and identify
web document pairs that are translations of each other. We release a new web
dataset consisting of over 392 million URL pairs from Common Crawl covering
documents in 8144 language pairs of which 137 pairs include English. In
addition to curating this massive dataset, we introduce baseline methods that
leverage cross-lingual representations to identify aligned documents based on
their textual content. Finally, we demonstrate the value of this parallel
documents dataset through a downstream task of mining parallel sentences and
measuring the quality of machine translations from models trained on this mined
data. Our objective in releasing this dataset is to foster new research in
cross-lingual NLP across a variety of low, medium, and high-resource languages.

利用 URL 标签指导对网络文章进行跨语言文档对齐，构建了一个大规模的网络文档数据集，使用该数据集通过基线模型进行文本内容的对齐，最终通过挖掘平行语句和测量基于挖掘数据训练的模型的机器翻译质量来证明了数据集的价值，旨在促进跨语言 NLP 研究的发展。