Language Models (LMs) pre-trained with self-supervision on large text corpora
have become the default starting point for developing models for various NLP
tasks. Once the pre-training corpus has been assembled, all data samples in the
corpus are treated with equal importance during LM pre-training. However, due
to varying levels of relevance and quality of data, equal importance to all the
data samples may not be the optimal choice. While data reweighting has been
explored in the context of task-specific supervised learning and LM
fine-tuning, model-driven reweighting for pre-training data has not been
explored. We fill this important gap and propose PRESENCE, a method for jointly
reweighting samples by leveraging self-influence (SI) scores as an indicator of
sample importance and pre-training. PRESENCE promotes novelty and stability for
model pre-training. Through extensive analysis spanning multiple model sizes,
datasets, and tasks, we present PRESENCE as an important first step in the
research direction of sample reweighting for pre-training language models.

我们提出了 PRESENCE 方法，通过利用自我影响（SI）得分作为样本重要性和预训练的指标来共同重新加权样本，从而促进模型预训练的新颖性和稳定性，填补了在预训练语言模型中采用模型驱动的样本重新加权的重要领域的空白。