This paper describes a web-based corpus of global language use with a focus
on how this corpus can be used for data-driven language mapping. First, the
corpus provides a representation of where national varieties of major languages
are used (e.g., English, Arabic, Russian) together with consistently collected
data for each variety. Second, the paper evaluates a language identification
model that supports more local languages with smaller sample sizes than
alternative off-the-shelf models. Improved language identification is essential
for moving beyond majority languages. Given the focus on language mapping, the
paper analyzes how well this digital language data represents actual
populations by (i) systematically comparing the corpus with demographic
ground-truth data and (ii) triangulating the corpus with an alternate
Twitter-based dataset. In total, the corpus contains 423 billion words
representing 148 languages (with over 1 million words from each language) and
158 countries (again with over 1 million words from each country), all
distilled from Common Crawl web data. The main contribution of this paper, in
addition to describing this publicly-available corpus, is to provide a
comprehensive analysis of the relationship between two sources of digital data
(the web and Twitter) as well as their connection to underlying populations.

本文描述了一个基于互联网语言使用的语料库，重点介绍如何利用这个语料库进行数据驱动的语言映射；通过提供有关全球语言使用的信息来支持本地化语种，并且通过对比人口统计学数据和 Twitter 数据来分析数字语言数据代表实际人口的程度。