Large generative language models have been very successful for English, but other languages lag behind due to data and computational limitations. We propose a method that may overcome these problems by adapting existing pre-trained language models to new languages. Specifically, we describe the adaptation of English GPT-2 to Italian and Dutch by retraining lexical embeddings without tuning the Transformer layers. As a result, we obtain lexical embeddings for Italian and Dutch that are aligned with the original English lexical embeddings and induce a bilingual lexicon from this alignment. Additionally, we show how to scale up complexity by transforming relearned lexical embeddings of GPT-2 small to the GPT-2 medium embedding space. This method minimises the amount of training and prevents losing information during adaptation that was learned by GPT-2. English GPT-2 models with relearned lexical embeddings can generate realistic sentences in Italian and Dutch, but on average these sentences are still identifiable as artificial by humans. Based on perplexity scores and human judgements, we find that generated sentences become more realistic with some additional full model finetuning, especially for Dutch. For Italian, we see that they are evaluated on par with sentences generated by a GPT-2 model fully trained from scratch. Our work can be conceived as a blueprint for training GPT-2s for other languages, and we provide a 'recipe' to do so.

本文提出了一种方法，将英语GPT-2模型通过重新训练词汇嵌入，适应意大利语和荷兰语，并通过增加复杂性来生成与原始英文嵌入对齐的意大利语和荷兰语词汇嵌入。该方法最大程度地减少了训练量，并且防止了在适应过程中由GPT-2学习的信息丢失，这使得英语GPT-2模型可以生成逼真的意大利语和荷兰语句子。虽然平均而言，这些句子仍然可被人们认为是人造的，但是它们与从头训练的GPT-2模型生成的句子相媲美。

全新如初：成功回收英文GPT-2制作其他语言模型