Adapting multilingual language models to specific languages can enhance both their efficiency and performance. In this study, we explore how modifying the vocabulary of a multilingual encoder model to better suit the Estonian language affects its downstream performance on the Named Entity Recognition (NER) task. The motivations for adjusting the vocabulary are twofold: practical benefits affecting the computational cost, such as reducing the input sequence length and the model size, and performance enhancements by tailoring the vocabulary to the particular language. We evaluate the effectiveness of two vocabulary adaptation approaches -- retraining the tokenizer and pruning unused tokens -- and assess their impact on the model's performance, particularly after continual training. While retraining the tokenizer degraded the performance of the NER task, suggesting that longer embedding tuning might be needed, we observed no negative effects on pruning.

本研究解决了多语言模型在特定语言，尤其是爱沙尼亚语中的适应性不足问题。通过调整词汇，我们探讨了重训练词汇器与删除未使用标记对模型性能的影响。结果显示，尽管重训练词汇器对命名实体识别任务的性能有所降低，但删除未使用标记并未产生负面影响，表明此方法可有效提高模型效率。

修剪或重训练：优化多语言模型在爱沙尼亚语中的词汇