Large language models (LLMs) have shown remarkable capabilities in many
languages beyond English. Yet, LLMs require more inference steps when
generating non-English text due to their reliance on English-centric
tokenizers, vocabulary, and pre-training data, resulting in higher usage costs
to non-English speakers. Vocabulary expansion with target language tokens is a
widely used cross-lingual vocabulary adaptation approach to remedy this issue.
Despite its effectiveness in inference speedup, the majority of previous work
has focused on high-resource settings assuming access to a substantial amount
of target language data to effectively initialize the embeddings of the new
tokens and adapt the LLM to the target language. However, vocabulary expansion
for LLMs in low-resource settings (i.e. languages and compute) has yet to be
explored. In this paper, we investigate sample-efficient adaptation strategies
from different angles, including target vocabulary size and initialization
methods, and the amount of target data available for adaptation. Extensive
experiments across typologically diverse languages, tasks and models show that
simpler heuristic-based embedding initialization is more efficient and robust
to changes in target vocabulary size and adaptation data in low-resource
settings, outperforming a popular random initialization and a more
sophisticated state-of-the-art approach that relies on external data and model.

通过在不同角度研究目标词汇规模、初始化方法以及可用于适应的目标数据量，我们在语料资源有限的环境中发现，基于简单启发式的词向量初始化方法更高效、更稳定，能够在目标词汇规模和适应数据变化时胜过常用的随机初始化和更复杂依赖外部数据和模型的最先进方法。

低资源跨语言迁移的词汇扩展

Vocabulary Expansion for Low-resource Cross-lingual Transfer

In this paper, we introduce UniBridge (Cross-Lingual Transfer Learning with
Optimized Embeddings and Vocabulary), a comprehensive approach developed to
improve the effectiveness of Cross-Lingual Transfer Learning, particularly in
languages with limited resources. Our approach tackles two essential elements
of a language model: the initialization of embeddings and the optimal
vocabulary size. Specifically, we propose a novel embedding initialization
method that leverages both lexical and semantic alignment for a language. In
addition, we present a method for systematically searching for the optimal
vocabulary size, ensuring a balance between model complexity and linguistic
coverage. Our experiments across multilingual datasets show that our approach
greatly improves the F1-Score in several languages. UniBridge is a robust and
adaptable solution for cross-lingual systems in various languages, highlighting
the significance of initializing embeddings and choosing the right vocabulary
size in cross-lingual environments.

UniBridge 是一个跨语言迁移学习的综合方法，通过优化嵌入和词汇表进行改进，在资源有限的语言中特别有效。该方法解决了语言模型的两个关键要素：嵌入的初始化和最优词汇表的选择。实验结果表明，UniBridge 显著提高了多种语言的 F1 得分，是跨语言环境中初始化嵌入和选择适当词汇表的可靠解决方案。

UniBridge：面向低资源语言的统一交叉语言迁移学习方法

UniBridge: A Unified Approach to Cross-Lingual Transfer Learning for  Low-Resource Languages

Using model weights pretrained on a high-resource language as a warm start
can reduce the need for data and compute to obtain high-quality language models
in low-resource languages. To accommodate the new language, the pretrained
vocabulary and embeddings need to be adapted. Previous work on embedding
initialization for such adapted vocabularies has mostly focused on monolingual
source models. In this paper, we investigate the multilingual source model
setting and propose FOCUS - Fast Overlapping Token Combinations Using
Sparsemax, a novel embedding initialization method that outperforms previous
work when adapting XLM-R. FOCUS represents newly added tokens as combinations
of tokens in the overlap of the pretrained and new vocabularies. The
overlapping tokens are selected based on semantic similarity in an auxiliary
token embedding space. Our implementation of FOCUS is publicly available on
GitHub.

使用高资源语言预训练的模型权重作为启动可以降低获取低资源语言高质量语言模型所需的数据和计算量，为了适应新语言，需要调整预先训练的词汇表和嵌入。本文研究了多语言源模型设置，提出了 FOCUS - Fast Overlapping Token Combinations Using Sparsemax 一种嵌入初始化方法，该方法在适应 XLM-R 时优于先前的方法，FOCUS 将新添加的令牌表示为预训练和新词汇表重叠的令牌组合。重叠的令牌是基于辅助令牌嵌入空间中的语义相似性选择的。