In the era of big data, the issue of data quality has become increasingly
prominent. One of the main challenges is the problem of duplicate data, which
can arise from repeated entry or the merging of multiple data sources. These
"dirty data" problems can significantly limit the effective application of big
data. To address the issue of data deduplication, we propose a pre-trained
deduplication model based on active learning, which is the first work that
utilizes active learning to address the problem of deduplication at the
semantic level. The model is built on a pre-trained Transformer and fine-tuned
to solve the deduplication problem as a sequence to classification task, which
firstly integrate the transformer with active learning into an end-to-end
architecture to select the most valuable data for deduplication model training,
and also firstly employ the R-Drop method to perform data augmentation on each
round of labeled data, which can reduce the cost of manual labeling and improve
the model's performance. Experimental results demonstrate that our proposed
model outperforms previous state-of-the-art (SOTA) for deduplicated data
identification, achieving up to a 28% improvement in Recall score on benchmark
datasets.

在大数据时代，数据质量问题变得越来越突出。我们提出了一种基于主动学习的预训练去重模型，该模型是首个将主动学习应用于语义级别去重问题的工作。实验结果表明，我们的模型在去重数据识别方面优于之前的最先进技术，基准数据集上的召回率得到了 28% 的提升。