Multilingual sentence encoders have seen much success in cross-lingual model
transfer for downstream NLP tasks. Yet, we know relatively little about the
properties of individual languages or the general patterns of linguistic
variation that they encode. We propose methods for probing sentence
representations from state-of-the-art multilingual encoders (LASER, M-BERT, XLM
and XLM-R) with respect to a range of typological properties pertaining to
lexical, morphological and syntactic structure. In addition, we investigate how
this information is distributed across all layers of the models. Our results
show interesting differences in encoding linguistic variation associated with
different pretraining strategies.

本文针对多种多语句子编码器，提出一种探究其对涵盖词汇、形态和句法结构的多种语言特性的表征方法，发现在不同的预训练策略下，编码不同语言特性的效果存在差异。

什么是语言无关性？探究多语句子编码器的类型学特性

What does it mean to be language-agnostic? Probing multilingual sentence  encoders for typological properties

How do typological properties such as word order and morphological case
marking affect the ability of neural sequence models to acquire the syntax of a
language? Cross-linguistic comparisons of RNNs' syntactic performance (e.g., on
subject-verb agreement prediction) are complicated by the fact that any two
languages differ in multiple typological properties, as well as by differences
in training corpus. We propose a paradigm that addresses these issues: we
create synthetic versions of English, which differ from English in one or more
typological parameters, and generate corpora for those languages based on a
parsed English corpus. We report a series of experiments in which RNNs were
trained to predict agreement features for verbs in each of those synthetic
languages. Among other findings, (1) performance was higher in
subject-verb-object order (as in English) than in subject-object-verb order (as
in Japanese), suggesting that RNNs have a recency bias; (2) predicting
agreement with both subject and object (polypersonal agreement) improves over
predicting each separately, suggesting that underlying syntactic knowledge
transfers across the two tasks; and (3) overt morphological case makes
agreement prediction significantly easier, regardless of word order.

研究了神经序列模型通过 typological properties 如词序和形态格标示来获取一种语言的语法的能力，针对这种种类繁多的语法特征和训练语料之间的复杂交互，作者使用了从英文中生成一些与英文在一个或多个语言学特性上不同的合成语言来避免影响，研究表明诸如 SVO 词序、多个格标示等特性可以有效提高合成语言的一致性效果。