Tokenisation is a core part of language models (LMs). It involves splitting a
character sequence into subwords which are assigned arbitrary indices before
being served to the LM. While typically lossless, however, this process may
lead to less sample efficient LM training: as it removes character-level
information, it could make it harder for LMs to generalise across similar
subwords, such as now and Now. We refer to such subwords as near duplicates. In
this paper, we study the impact of near duplicate subwords on LM training
efficiency. First, we design an experiment that gives us an upper bound to how
much we should expect a model to improve if we could perfectly generalise
across near duplicates. We do this by duplicating each subword in our LM's
vocabulary, creating perfectly equivalent classes of subwords. Experimentally,
we find that LMs need roughly 17% more data when trained in a fully duplicated
setting. Second, we investigate the impact of naturally occurring near
duplicates on LMs. Here, we see that merging them considerably hurts LM
performance. Therefore, although subword duplication negatively impacts LM
training efficiency, naturally occurring near duplicates may not be as similar
as anticipated, limiting the potential for performance improvements.

研究论文中，我们主要探讨了词元化对语言模型训练效率的影响，发现在完全重复设置下，语言模型训练需要大约多出 17% 的数据量。此外，研究还发现自然存在的近义词对语言模型的性能产生了负面影响，限制了表现的潜力。

关于语言模型中（近似）重复的子词的影响

On the Effect of (Near) Duplicate Subwords in Language Modelling

Identifying near duplicates within large, noisy text corpora has a myriad of
applications that range from de-duplicating training datasets, reducing privacy
risk, and evaluating test set leakage, to identifying reproduced news articles
and literature within large corpora. Across these diverse applications, the
overwhelming majority of work relies on N-grams. Limited efforts have been made
to evaluate how well N-gram methods perform, in part because it is unclear how
one could create an unbiased evaluation dataset for a massive corpus. This
study uses the unique timeliness of historical news wires to create a 27,210
document dataset, with 122,876 positive duplicate pairs, for studying
noise-robust de-duplication. The time-sensitivity of news makes comprehensive
hand labelling feasible - despite the massive overall size of the corpus - as
duplicates occur within a narrow date range. The study then develops and
evaluates a range of de-duplication methods: hashing and N-gram overlap (which
predominate in the literature), a contrastively trained bi-encoder, and a
re-rank style approach combining a bi- and cross-encoder. The neural approaches
significantly outperform hashing and N-gram overlap. We show that the
bi-encoder scales well, de-duplicating a 10 million article corpus on a single
GPU card in a matter of hours. The public release of our NEWS-COPY
de-duplication dataset will facilitate further research and applications.

使用历史新闻电线的独特时效性创建了一个包含 27,210 个文件，122,876 个正重复对的数据集，评估了区分训练的双编码器和组合双编码器和交叉编码器的范围，结果表明神经方法显着优于哈希和 N-gram 重叠，并且 bi-encoder 在单个 GPU 卡上的可伸缩性很高。