Training monolingual language models for low and mid-resource languages is
made challenging by limited and often inadequate pretraining data. In this
study, we propose a novel model conversion strategy to address this issue,
adapting high-resources monolingual language models to a new target language.
By generalizing over a word translation dictionary encompassing both the source
and target languages, we map tokens from the target tokenizer to semantically
similar tokens from the source language tokenizer. This one-to-many token
mapping improves tremendously the initialization of the embedding table for the
target language. We conduct experiments to convert high-resource models to mid-
and low-resource languages, namely Dutch and Frisian. These converted models
achieve a new state-of-the-art performance on these languages across all sorts
of downstream tasks. By reducing significantly the amount of data and time
required for training state-of-the-art models, our novel model conversion
strategy has the potential to benefit many languages worldwide.

通过模型转换策略将高资源单语言模型转化为新的目标语言，从而在低资源语言和中等资源语言上实现了新的最先进性能。

Tik-to-Tok：一个逐令牌翻译语言模型的嵌入初始化策略用于有效语言适应

Tik-to-Tok: Translating Language Models One Token at a Time: An  Embedding Initialization Strategy for Efficient Language Adaptation

In this work, we present a simple and elegant approach to language modeling
for bilingual code-switched text. Since code-switching is a blend of two or
more different languages, a standard bilingual language model can be improved
upon by using structures of the monolingual language models. We propose a novel
technique called dual language models, which involves building two
complementary monolingual language models and combining them using a
probabilistic model for switching between the two. We evaluate the efficacy of
our approach using a conversational Mandarin-English speech corpus. We prove
the robustness of our model by showing significant improvements in perplexity
measures over the standard bilingual language model without the use of any
external information. Similar consistent improvements are also reflected in
automatic speech recognition error rates.

本文提出了一种针对双语混杂文本的语言模型的新方法，即双语言模型，该方法使用单一语言模型的结构来改进标准的双语言模型，并将两个互补的单一语言模型结合在一起，以概率切换模型进行切换。我们使用一种会话式汉英语音语料库评估了该方法的有效性，并证明了我们的模型的鲁棒性，表明在不使用任何外部信息的情况下，在困惑度措施和自动语音识别错误率方面都会显著提高。