This paper empirically investigates the relationship between subword
vocabulary size and the performance of large language models (LLMs) to provide
insights on how to define the vocabulary size. Experimental results show that
larger vocabulary sizes lead to better performance in LLMs. Moreover, we
consider a continual training scenario where a pre-trained language model is
trained on a different target language. We introduce a simple method to use a
new vocabulary instead of the pre-defined one. We show that using the new
vocabulary outperforms the model with the vocabulary used in pre-training.

本文通过实证研究了子词词汇大小与大型语言模型（LLM）性能之间的关系，以提供关于如何定义词汇大小的洞见。实验结果表明，更大的词汇大小可提高 LLM 的性能。此外，我们考虑了一个持续训练的情景，其中一个预训练的语言模型在不同的目标语言上进行训练。我们引入了一种简单的方法，使用新的词汇代替预定义的词汇。我们证明使用新词汇优于使用预训练词汇的模型。

大词汇量提升大型语言模型

Large Vocabulary Size Improves Large Language Models

To understand the growing phenomena of new vocabulary on nationwide online
social media, we analyzed monthly word count time series extracted from
approximately 1 billion Japanese blog articles from 2007 to 2019. In
particular, we first introduced the extended logistic equation by adding one
parameter to the original equation and showed that the model can consistently
reproduce various patterns of actual growth curves, such as the logistic
function, linear growth, and finite-time divergence. Second, by analyzing the
model parameters, we found that the typical growth pattern is not only a
logistic function, which often appears in various complex systems, but also a
nontrivial growth curve that starts with an exponential function and
asymptotically approaches a power function without a steady state. Furthermore,
we observed a connection between the functional form of growth and the
peak-out. Finally, we showed that the proposed model and statistical properties
are also valid for Google Trends data (English, French, Spanish, and Japanese),
which is a time series of the nationwide popularity of search queries.

通过分析日本博客文章并使用扩展的 Logistic 方程式，研究了新词汇在全国性在线社交媒体上的增长模式和趋势，并发现了新词汇增长的模式与传统复杂系统的逻辑函数有所不同，同时对 Google Trends 数据进行了验证。

在线媒体字数增长的 Logistic 方程微小扩展：对社会增长现象多样性的参数描述

A minor extension of the logistic equation for growth of word counts on online media: Parametric description of diversity of growth phenomena in society