Despite an ever growing number of word representation models introduced for a
large number of languages, there is a lack of a standardized technique to
provide insights into what is captured by these models. Such insights would
help the community to get an estimate of the downstream task performance, as
well as to design more informed neural architectures, while avoiding extensive
experimentation which requires substantial computational resources not all
researchers have access to. A recent development in NLP is to use simple
classification tasks, also called probing tasks, that test for a single
linguistic feature such as part-of-speech. Existing studies mostly focus on
exploring the linguistic information encoded by the continuous representations
of English text. However, from a typological perspective the morphologically
poor English is rather an outlier: the information encoded by the word order
and function words in English is often stored on a morphological level in other
languages. To address this, we introduce 15 type-level probing tasks such as
case marking, possession, word length, morphological tag count and pseudoword
identification for 24 languages. We present a reusable methodology for creation
and evaluation of such tests in a multilingual setting. We then present
experiments on several diverse multilingual word embedding models, in which we
relate the probing task performance for a diverse set of languages to a range
of five classic NLP tasks: POS-tagging, dependency parsing, semantic role
labeling, named entity recognition and natural language inference. We find that
a number of probing tests have significantly high positive correlation to the
downstream tasks, especially for morphologically rich languages. We show that
our tests can be used to explore word embeddings or black-box neural models for
linguistic cues in a multilingual setting.

该研究介绍了在 24 种语言中引入 15 种类型级别的探究任务，测试诸如情况标记、单词长度、形态标记计数和伪词识别等特定语法特征和语言提示的分类任务，以便探索词嵌入或黑盒神经模型的多语言语言线索，发现许多探究测试具有与子任务的显着高正相关性，特别是对于形态丰富的语言。