To what extent can neural network models learn generalizations about language
structure, and how do we find out what they have learned? We explore these
questions by training neural models for a range of natural language processing
tasks on a massively multilingual dataset of Bible translations in 1295
languages. The learned language representations are then compared to existing
typological databases as well as to a novel set of quantitative syntactic and
morphological features obtained through annotation projection. We conclude that
some generalizations are surprisingly close to traditional features from
linguistic typology, but that most of our models, as well as those of previous
work, do not appear to have made linguistically meaningful generalizations.
Careful attention to details in the evaluation turns out to be essential to
avoid false positives. Furthermore, to encourage continued work in this field,
we release several resources covering most or all of the languages in our data:
(i) multiple sets of language representations, (ii) multilingual word
embeddings, (iii) projected and predicted syntactic and morphological features,
(iv) software to provide linguistically sound evaluations of language
representations.

在 1295 种语言的大规模多语言数据集中训练神经网络模型，以研究神经模型对于语言结构所能学习到的广义化程度，发现神经网络模型在语言结构的广义化方面的表现并不好，并会存在误报，但有些表现仍然与语言学中的传统特征非常接近，为了鼓励在这个领域的持续研究，我们发布了多个资源，包括语言表示的多重集合、多语言单词嵌入、映射和预测的语法和形态特征以及提供语言表征的具有语言学意义的评估软件。