We evaluate whether BERT, a widely used neural network for sentence
processing, acquires an inductive bias towards forming structural
generalizations through pretraining on raw data. We conduct four experiments
testing its preference for structural vs. linear generalizations in different
structure-dependent phenomena. We find that BERT makes a structural
generalization in 3 out of 4 empirical domains---subject-auxiliary inversion,
reflexive binding, and verb tense detection in embedded clauses---but makes a
linear generalization when tested on NPI licensing. We argue that these results
are the strongest evidence so far from artificial learners supporting the
proposition that a structural bias can be acquired from raw data. If this
conclusion is correct, it is tentative evidence that some linguistic universals
can be acquired by learners without innate biases. However, the precise
implications for human language acquisition are unclear, as humans learn
language from significantly less data than BERT.

通过针对不同结构相关现象的四个实验，我们评估了 BERT 在原始数据的预训练过程中是否获得了形成结构化概括的归纳偏差，并发现 BERT 在实体 - 助动词倒装，反身代词，以及嵌套从句中动词时态检测中进行了结构化概括，但在 NPI 许可证检测中进行了线性概括，这是迄今为止最有力的支持从原始数据中获取结构性偏差的人工学习器研究结果之一。

神经网络是否可以从原始语言数据中获得结构偏见？

Can neural networks acquire a structural bias from raw linguistic data?

Recent work has found evidence that Multilingual BERT (mBERT), a
transformer-based multilingual masked language model, is capable of zero-shot
cross-lingual transfer, suggesting that some aspects of its representations are
shared cross-lingually. To better understand this overlap, we extend recent
work on finding syntactic trees in neural networks' internal representations to
the multilingual setting. We show that subspaces of mBERT representations
recover syntactic tree distances in languages other than English, and that
these subspaces are approximately shared across languages. Motivated by these
results, we present an unsupervised analysis method that provides evidence
mBERT learns representations of syntactic dependency labels, in the form of
clusters which largely agree with the Universal Dependencies taxonomy. This
evidence suggests that even without explicit supervision, multilingual masked
language models learn certain linguistic universals.

本研究针对多语言 BERT 的序列到序列建模进行实验，证实其不需要训练就能够实现跨语言的转化，并提出一种无监督的分析方法，证明 Multilingual BERT 可以学习到一定的语言普遍规律。

在多语言 BERT 中发现通用语法关系

Finding Universal Grammatical Relations in Multilingual BERT

We train one multilingual model for dependency parsing and use it to parse
sentences in several languages. The parsing model uses (i) multilingual word
clusters and embeddings; (ii) token-level language information; and (iii)
language-specific features (fine-grained POS tags). This input representation
enables the parser not only to parse effectively in multiple languages, but
also to generalize across languages based on linguistic universals and
typological similarities, making it more effective to learn from limited
annotations. Our parser's performance compares favorably to strong baselines in
a range of data scenarios, including when the target language has a large
treebank, a small treebank, or no treebank for training.

使用多语言模型进行依赖解析，通过使用多语言单词聚类和嵌入，令解析器能够在多种语言中进行有效解析，同时基于语言普遍规律和类型相似性实现跨语言推断，从而使其能够更有效地从有限的注释中进行学习。