Recent work on the interpretability of deep neural language models has
concluded that many properties of natural language syntax are encoded in their
representational spaces. However, such studies often suffer from limited scope
by focusing on a single language and a single linguistic formalism. In this
study, we aim to investigate the extent to which the semblance of syntactic
structure captured by language models adheres to a surface-syntactic or deep
syntactic style of analysis, and whether the patterns are consistent across
different languages. We apply a probe for extracting directed dependency trees
to BERT and ELMo models trained on 13 different languages, probing for two
different syntactic annotation styles: Universal Dependencies (UD),
prioritizing deep syntactic relations, and Surface-Syntactic Universal
Dependencies (SUD), focusing on surface structure. We find that both models
exhibit a preference for UD over SUD - with interesting variations across
languages and layers - and that the strength of this preference is correlated
with differences in tree shape.

研究深度神经语言模型的可解释性，通过在多种语言模型中应用基于深度句法和表层句法的分析方法，考察其对语法结构的捕获程度以及不同语言间的一致性关系，并发现该语言模型普遍倾向 “Universal Dependencies” 语法形式，并且这种倾向的强度与树形状的差异有关。