This paper investigates the problem of learning cross-lingual representations in a contextual space. We propose Cross-Lingual BERT Transformation (CLBT), a simple and efficient approach to generate cross-lingual contextualized word embeddings based on publicly available pre-trained BERT models (Devlin et al., 2018). In this approach, a linear transformation is learned from contextual word alignments to align the contextualized embeddings independently trained in different languages. We demonstrate the effectiveness of this approach on zero-shot cross-lingual transfer parsing. Experiments show that our embeddings substantially outperform the previous state-of-the-art that uses static embeddings. We further compare our approach with XLM (Lample and Conneau, 2019), a recently proposed cross-lingual language model trained with massive parallel data, and achieve highly competitive results.

本文利用公共预训练的BERT模型提出了学习跨语言表征在上下文空间中的新方法——Cross-Lingual BERT Transformation (CLBT)，其中通过线性变换将训练不同语言的上下文化嵌入与上下文词对齐并结合实验证明了该方法可在无需训练就能在跨语言解析的领域中取得优势，其效果明显优于使用静态嵌入的之前的最先进技术，比较证明了本方法与最近被提出的使用大规模平行数据训练的交叉语言模型XLM的竞争力。

零-shot依赖句法分析的跨语言BERT转换