Large language models' (LLMs) abilities are drawn from their pretraining
data, and model development begins with data curation. However, decisions
around what data is retained or removed during this initial stage is
under-scrutinized. In our work, we ground web text, which is a popular
pretraining data source, to its social and geographic contexts. We create a new
dataset of 10.3 million self-descriptions of website creators, and extract
information about who they are and where they are from: their topical
interests, social roles, and geographic affiliations. Then, we conduct the
first study investigating how ten "quality" and English language identification
(langID) filters affect webpages that vary along these social dimensions. Our
experiments illuminate a range of implicit preferences in data curation: we
show that some quality classifiers act like topical domain filters, and langID
can overlook English content from some regions of the world. Overall, we hope
that our work will encourage a new line of research on pretraining data
curation practices and its social implications.

大型语言模型的能力来源于它们的预训练数据，而模型开发始于数据筛选。我们的研究基于网络文本，将其连接到社交和地理背景，并创建了一个包含 1030 万个网站创作者自我描述的新数据集，提取了关于他们的兴趣、社交角色和地理隶属的信息。然后，我们进行了第一项研究，调查了十种 “质量” 和英语语言标识（langID）过滤器对这些社交维度变化的网页的影响。我们的实验揭示了数据筛选中一系列的隐含偏好：我们发现一些质量分类器像专题领域过滤器一样起到作用，而 langID 可能会忽视世界上某些地区的英语内容。总体而言，我们希望我们的工作能够促进关于预训练数据筛选实践及其社会影响的新的研究方向。