Although Shapley values have been shown to be highly effective for
identifying harmful training instances, dataset size and model complexity
constraints limit the ability to apply Shapley-based data valuation to
fine-tuning large pre-trained language models. To address this, we propose
TS-DShapley, an algorithm that reduces computational cost of Shapley-based data
valuation through: 1) an efficient sampling-based method that aggregates
Shapley values computed from subsets for valuation of the entire training set,
and 2) a value transfer method that leverages value information extracted from
a simple classifier trained using representations from the target language
model. Our experiments applying TS-DShapley to select data for fine-tuning
BERT-based language models on benchmark natural language understanding (NLU)
datasets show that TS-DShapley outperforms existing data selection methods.
Further, TS-DShapley can filter fine-tuning data to increase language model
performance compared to training with the full fine-tuning dataset.

提出了一种名为 TS-DShapley 的算法，通过一种高效的基于采样的方法和一种数据价值信息的传递方法，大大减少了 Shapley 基于数据评估的计算成本，可以对大型预训练语言模型进行微调，并在基准自然语言理解（NLU）数据集上提升了语言模型的性能。