Language models increasingly rely on massive web dumps for diverse text data.
However, these sources are rife with undesirable content. As such, resources
like Wikipedia, books, and newswire often serve as anchors for automatically
selecting web text most suitable for language modeling, a process typically
referred to as quality filtering. Using a new dataset of U.S. high school
newspaper articles -- written by students from across the country -- we
investigate whose language is preferred by the quality filter used for GPT-3.
We find that newspapers from larger schools, located in wealthier, educated,
and urban ZIP codes are more likely to be classified as high quality. We then
demonstrate that the filter's measurement of quality is unaligned with other
sensible metrics, such as factuality or literary acclaim. We argue that
privileging any corpus as high quality entails a language ideology, and more
care is needed to construct training corpora for language models, with better
transparency and justification for the inclusion or exclusion of various texts.

本文使用美国高中学生撰写的报纸文章，研究了 GPT-3 质量筛选器偏向于哪些学校、地理区域和社会阶层的语言。研究发现质量筛选器的质量测量与事实和文学赞誉等合理指标不一致，强调任何语料库为高质量都涉及到语言意识形态，需要更谨慎地构建语言模型培训语料库，更好地明确各种文本的包含或排除的透明度和理由。