Cross-lingual adaptation with multilingual pre-trained language models
(mPTLMs) mainly consists of two lines of works: zero-shot approach and
translation-based approach, which have been studied extensively on the
sequence-level tasks. We further verify the efficacy of these cross-lingual
adaptation approaches by evaluating their performances on more fine-grained
sequence tagging tasks. After re-examining their strengths and drawbacks, we
propose a novel framework to consolidate the zero-shot approach and the
translation-based approach for better adaptation performance. Instead of simply
augmenting the source data with the machine-translated data, we tailor-make a
warm-up mechanism to quickly update the mPTLMs with the gradients estimated on
a few translated data. Then, the adaptation approach is applied to the refined
parameters and the cross-lingual transfer is performed in a warm-start way. The
experimental results on nine target languages demonstrate that our method is
beneficial to the cross-lingual adaptation of various sequence tagging tasks.

研究了零样本学习和基于翻译的方法在预训练语言模型（mPTLMs）的跨语言适应性中的应用，提出了一种新的框架来整合这两种方法，通过定制一个预热机制来快速更新 mPTLMs，应用于细化的参数上进行跨语言适应性的实验结果表明，我们的方法有助于各种序列标注任务的跨语言适应性。