Multilingual sentence encoders have seen much success in cross-lingual model
transfer for downstream NLP tasks. Yet, we know relatively little about the
properties of individual languages or the general patterns of linguistic
variation that they encode. We propose methods for probing sentence
representations from state-of-the-art multilingual encoders (LASER, M-BERT, XLM
and XLM-R) with respect to a range of typological properties pertaining to
lexical, morphological and syntactic structure. In addition, we investigate how
this information is distributed across all layers of the models. Our results
show interesting differences in encoding linguistic variation associated with
different pretraining strategies.

本文针对多种多语句子编码器，提出一种探究其对涵盖词汇、形态和句法结构的多种语言特性的表征方法，发现在不同的预训练策略下，编码不同语言特性的效果存在差异。

什么是语言无关性？探究多语句子编码器的类型学特性

What does it mean to be language-agnostic? Probing multilingual sentence  encoders for typological properties

In this paper, we show that Multilingual BERT (M-BERT), released by Devlin et
al. (2018) as a single language model pre-trained from monolingual corpora in
104 languages, is surprisingly good at zero-shot cross-lingual model transfer,
in which task-specific annotations in one language are used to fine-tune the
model for evaluation in another language. To understand why, we present a large
number of probing experiments, showing that transfer is possible even to
languages in different scripts, that transfer works best between typologically
similar languages, that monolingual corpora can train models for
code-switching, and that the model can find translation pairs. From these
results, we can conclude that M-BERT does create multilingual representations,
but that these representations exhibit systematic deficiencies affecting
certain language pairs.

研究表明，在零 - shot 跨语言模型转移方面，多语言 BERT（M-BERT）表现出惊人的性能，经过大量探究实验，证明转移甚至可以到不同文字的语言中，但它们会因特定的语言对而表现出系统缺陷。

Multilingual BERT 的多语言能力如何？

How multilingual is Multilingual BERT?

Cross-lingual model transfer is a compelling and popular method for
predicting annotations in a low-resource language, whereby parallel corpora
provide a bridge to a high-resource language and its associated annotated
corpora. However, parallel data is not readily available for many languages,
limiting the applicability of these approaches. We address these drawbacks in
our framework which takes advantage of cross-lingual word embeddings trained
solely on a high coverage bilingual dictionary. We propose a novel neural
network model for joint training from both sources of data based on
cross-lingual word embeddings, and show substantial empirical improvements over
baseline techniques. We also propose several active learning heuristics, which
result in improvements over competitive benchmark methods.

本研究提出了一种新颖的基于跨语言词向量的神经网络模型，通过高覆盖的双语词典训练，利用跨语言模型转移的方法，针对低资源语言的注释预测问题提出了解决方案，并通过多种主动学习启发式方法，提升了该方法的性能。