This paper introduces RETSim (Resilient and Efficient Text Similarity), a lightweight, multilingual deep learning model trained to produce robust metric embeddings for near-duplicate text retrieval, clustering, and dataset deduplication tasks. We demonstrate that RETSim is significantly more robust and accurate than MinHash and neural text embeddings, achieving new state-of-the-art performance on dataset deduplication, adversarial text retrieval benchmarks, and spam clustering tasks. We also introduce the W4NT3D benchmark (Wiki-40B 4dversarial Near-T3xt Dataset) for evaluating multilingual, near-duplicate text retrieval capabilities under adversarial settings. RETSim and the W4NT3D benchmark are open-sourced under the MIT License at https://github.com/google/unisim.

这篇论文介绍了 RETSim，一种轻量级、多语言深度学习模型，用于生成鲁棒的度量嵌入，用于近似重复文本检索、聚类和数据集去重任务。我们证明了 RETSim 在数据集去重、对抗性文本检索基准和垃圾邮件聚类任务上的鲁棒性和准确性明显优于 MinHash 和神经文本嵌入，实现了新的技术水平。我们还介绍了 W4NT3D 基准（Wiki-40B 对抗近似文本数据集），用于评估对抗性环境下多语言近似重复文本检索的能力。RETSim 和 W4NT3D 基准在 MIT 许可下开源。

RETSim：文本相似度的韧性与高效性