This paper investigates the impact of corpus creation decisions on large
multi-lingual geographic web corpora. Beginning with a 427 billion word corpus
derived from the Common Crawl, three methods are used to improve the quality of
sub-corpora representing specific language-country pairs like New Zealand
English: (i) the agreement of independent language identification systems, (ii)
hash-based deduplication, and (iii) location-specific outlier detection. The
impact of each of these steps is then evaluated at the language level and the
country level by using corpus similarity measures to compare each resulting
corpus with baseline data sets. The goal is to understand the impact of
upstream data cleaning decisions on downstream corpora with a specific focus on
under-represented languages and populations. The evaluation shows that the
validity of sub-corpora is improved with each stage of cleaning but that this
improvement is unevenly distributed across languages and populations. This
result shows how standard corpus creation techniques can accidentally exclude
under-represented populations.

本文研究了语料库创建决策对大型多语言地理 Web 语料库的影响。通过从 Common Crawl 获取的 4270 亿个词的语料库，使用三种方法改善代表特定语言国家对 (如新西兰英语) 的子语料库质量：(i) 独立语言识别系统的一致性，(ii) 基于哈希的去重，以及 (iii) 特定位置的异常检测。然后，通过使用语料库相似度度量将每个结果语料库与基准数据集进行比较，评估每个步骤对语言级别和国家级别的影响。目标是了解上游数据清理决策对下游语料库的影响，重点关注代表性不足的语言和人口。评估结果表明，每个清理阶段都提高了子语料库的有效性，但这种改进在语言和人口之间分布不均匀。该研究结果显示了标准语料库创建技术可能会无意中排除代表性不足的人口。