Incorporating stronger syntactic biases into neural language models (LMs) is
a long-standing goal, but research in this area often focuses on modeling
English text, where constituent treebanks are readily available. Extending
constituent tree-based LMs to the multilingual setting, where dependency
treebanks are more common, is possible via dependency-to-constituency
conversion methods. However, this raises the question of which tree formats are
best for learning the model, and for which languages. We investigate this
question by training recurrent neural network grammars (RNNGs) using various
conversion methods, and evaluating them empirically in a multilingual setting.
We examine the effect on LM performance across nine conversion methods and five
languages through seven types of syntactic tests. On average, the performance
of our best model represents a 19 \% increase in accuracy over the worst choice
across all languages. Our best model shows the advantage over
sequential/overparameterized LMs, suggesting the positive effect of syntax
injection in a multilingual setting. Our experiments highlight the importance
of choosing the right tree formalism, and provide insights into making an
informed decision.

通过依存句法树到常量句法树的转换，使用递归神经网络语法模型（RNNGs）进行训练，并在多语言环境下进行了实证评估，来探讨在学习模型时哪种树形式最佳，对哪些语言最适用，并通过七种类型的句法测试比较了 9 种转换方法和 5 种语言的语言模型表现，研究结果显示，最佳模型相对于最差模型在所有语言中的精确度提高了 19％，也证明了在多语言环境中注入语法的积极作用。

通过依赖树转换进行多语言语法感知语言建模

Multilingual Syntax-aware Language Modeling through Dependency Tree Conversion

Textual representation learners trained on large amounts of data have
achieved notable success on downstream tasks; intriguingly, they have also
performed well on challenging tests of syntactic competence. Given this
success, it remains an open question whether scalable learners like BERT can
become fully proficient in the syntax of natural language by virtue of data
scale alone, or whether they still benefit from more explicit syntactic biases.
To answer this question, we introduce a knowledge distillation strategy for
injecting syntactic biases into BERT pretraining, by distilling the
syntactically informative predictions of a hierarchical---albeit harder to
scale---syntactic language model. Since BERT models masked words in
bidirectional context, we propose to distill the approximate marginal
distribution over words in context from the syntactic LM. Our approach reduces
relative error by 2-21% on a diverse set of structured prediction tasks,
although we obtain mixed results on the GLUE benchmark. Our findings
demonstrate the benefits of syntactic biases, even in representation learners
that exploit large amounts of data, and contribute to a better understanding of
where syntactic biases are most helpful in benchmarks of natural language
understanding.

本文研究了文本表示学习器在句法表现上的优越性和在自然语言句法中的应用，提出了将句法模型的预测注入 BERT 的知识蒸馏策略，并表明在一系列结构预测任务中，这种方法能够显著地减少相对误差。