Language Models (LMs) pre-trained with self-supervision on large text corpora
have become the default starting point for developing models for various NLP
tasks. Once the pre-training corpus has been assembled, all data samples in the
corpus are treated with equal importance during LM pre-training. However, due
to varying levels of relevance and quality of data, equal importance to all the
data samples may not be the optimal choice. While data reweighting has been
explored in the context of task-specific supervised learning and LM
fine-tuning, model-driven reweighting for pre-training data has not been
explored. We fill this important gap and propose PRESENCE, a method for jointly
reweighting samples by leveraging self-influence (SI) scores as an indicator of
sample importance and pre-training. PRESENCE promotes novelty and stability for
model pre-training. Through extensive analysis spanning multiple model sizes,
datasets, and tasks, we present PRESENCE as an important first step in the
research direction of sample reweighting for pre-training language models.

我们提出了 PRESENCE 方法，通过利用自我影响（SI）得分作为样本重要性和预训练的指标来共同重新加权样本，从而促进模型预训练的新颖性和稳定性，填补了在预训练语言模型中采用模型驱动的样本重新加权的重要领域的空白。

自我引导的数据重新加权用于语言模型预训练

Self-Influence Guided Data Reweighting for Language Model Pre-training

Increasingly larger datasets have become a standard ingredient to advancing
the state of the art in NLP. However, data quality might have already become
the bottleneck to unlock further gains. Given the diversity and the sizes of
modern datasets, standard data filtering is not straight-forward to apply,
because of the multifacetedness of the harmful data and elusiveness of
filtering rules that would generalize across multiple tasks. We study the
fitness of task-agnostic self-influence scores of training examples for data
cleaning, analyze their efficacy in capturing naturally occurring outliers, and
investigate to what extent self-influence based data cleaning can improve
downstream performance in machine translation, question answering and text
classification, building up on recent approaches to self-influence calculation
and automated curriculum learning.

本文研究使用任务不可知的自我影响分数对训练数据进行清洗的有效性，通过分析其在捕捉自然异常值方面的功效来调查自我影响数据清洗对机器翻译、问答和文本分类等任务的改进程度，利用自我影响计算的最新方法和自动课程学习作为基础。