Large language models' (LLMs) abilities are drawn from their pretraining
data, and model development begins with data curation. However, decisions
around what data is retained or removed during this initial stage is
under-scrutinized. In our work, we ground web text, which is a popular
pretraining data source, to its social and geographic contexts. We create a new
dataset of 10.3 million self-descriptions of website creators, and extract
information about who they are and where they are from: their topical
interests, social roles, and geographic affiliations. Then, we conduct the
first study investigating how ten "quality" and English language identification
(langID) filters affect webpages that vary along these social dimensions. Our
experiments illuminate a range of implicit preferences in data curation: we
show that some quality classifiers act like topical domain filters, and langID
can overlook English content from some regions of the world. Overall, we hope
that our work will encourage a new line of research on pretraining data
curation practices and its social implications.

大型语言模型的能力来源于它们的预训练数据，而模型开发始于数据筛选。我们的研究基于网络文本，将其连接到社交和地理背景，并创建了一个包含 1030 万个网站创作者自我描述的新数据集，提取了关于他们的兴趣、社交角色和地理隶属的信息。然后，我们进行了第一项研究，调查了十种 “质量” 和英语语言标识（langID）过滤器对这些社交维度变化的网页的影响。我们的实验揭示了数据筛选中一系列的隐含偏好：我们发现一些质量分类器像专题领域过滤器一样起到作用，而 langID 可能会忽视世界上某些地区的英语内容。总体而言，我们希望我们的工作能够促进关于预训练数据筛选实践及其社会影响的新的研究方向。

关于我：使用网页中的自我描述来记录英语预训练数据过滤的效果

AboutMe: Using Self-Descriptions in Webpages to Document the Effects of  English Pretraining Data Filters

Relational facts are an important component of human knowledge, which are
hidden in vast amounts of text. In order to extract these facts from text,
people have been working on relation extraction (RE) for years. From early
pattern matching to current neural networks, existing RE methods have achieved
significant progress. Yet with explosion of Web text and emergence of new
relations, human knowledge is increasing drastically, and we thus require
"more" from RE: a more powerful RE system that can robustly utilize more data,
efficiently learn more relations, easily handle more complicated context, and
flexibly generalize to more open domains. In this paper, we look back at
existing RE methods, analyze key challenges we are facing nowadays, and show
promising directions towards more powerful RE. We hope our view can advance
this field and inspire more efforts in the community.

本文回顾了现有的关系抽取方法，分析了当前面临的关键挑战，并展示了通向更强大的关系抽取的有前途的方向。

更多数据、更多关系、更多上下文和更多开放：关系抽取综述与展望

More Data, More Relations, More Context and More Openness: A Review and  Outlook for Relation Extraction

Argument mining is a core technology for automating argument search in large
document collections. Despite its usefulness for this task, most current
approaches to argument mining are designed for use only with specific text
types and fall short when applied to heterogeneous texts. In this paper, we
propose a new sentential annotation scheme that is reliably applicable by crowd
workers to arbitrary Web texts. We source annotations for over 25,000 instances
covering eight controversial topics. The results of cross-topic experiments
show that our attention-based neural network generalizes best to unseen topics
and outperforms vanilla BiLSTM models by 6% in accuracy and 11% in F-score.

本文提出了一种新的句子注释方案，用于在任意网络文本上进行争议搜索的分析，得到的神经网络模型在准确度和 F1 分数上均优于传统的双向 LSTM 模型。