Identifying near duplicates within large, noisy text corpora has a myriad of
applications that range from de-duplicating training datasets, reducing privacy
risk, and evaluating test set leakage, to identifying reproduced news articles
and literature within large corpora. Across these diverse applications, the
overwhelming majority of work relies on N-grams. Limited efforts have been made
to evaluate how well N-gram methods perform, in part because it is unclear how
one could create an unbiased evaluation dataset for a massive corpus. This
study uses the unique timeliness of historical news wires to create a 27,210
document dataset, with 122,876 positive duplicate pairs, for studying
noise-robust de-duplication. The time-sensitivity of news makes comprehensive
hand labelling feasible - despite the massive overall size of the corpus - as
duplicates occur within a narrow date range. The study then develops and
evaluates a range of de-duplication methods: hashing and N-gram overlap (which
predominate in the literature), a contrastively trained bi-encoder, and a
re-rank style approach combining a bi- and cross-encoder. The neural approaches
significantly outperform hashing and N-gram overlap. We show that the
bi-encoder scales well, de-duplicating a 10 million article corpus on a single
GPU card in a matter of hours. The public release of our NEWS-COPY
de-duplication dataset will facilitate further research and applications.

使用历史新闻电线的独特时效性创建了一个包含 27,210 个文件，122,876 个正重复对的数据集，评估了区分训练的双编码器和组合双编码器和交叉编码器的范围，结果表明神经方法显着优于哈希和 N-gram 重叠，并且 bi-encoder 在单个 GPU 卡上的可伸缩性很高。