A line of work on Transformer-based language models such as BERT has
attempted to use syntactic inductive bias to enhance the pretraining process,
on the theory that building syntactic structure into the training process
should reduce the amount of data needed for training. But such methods are
often tested for high-resource languages such as English. In this work, we
investigate whether these methods can compensate for data sparseness in
low-resource languages, hypothesizing that they ought to be more effective for
low-resource languages. We experiment with five low-resource languages: Uyghur,
Wolof, Maltese, Coptic, and Ancient Greek. We find that these syntactic
inductive bias methods produce uneven results in low-resource settings, and
provide surprisingly little benefit in most cases.

一项关于基于 Transformer 的语言模型（如 BERT）的研究尝试使用语法归纳偏置来增强预训练过程，理论上通过将语法结构融入训练过程可以降低训练所需的数据量。但此类方法通常在高资源语言（如英语）上进行测试。在这项研究中，我们调查了这些方法是否可以弥补低资源语言中的数据稀疏性，研究假设它们在低资源语言中应该更加有效。我们对五种低资源语言进行了实验：维吾尔语、沃洛夫语、马耳他语、科普特语和古希腊语。我们发现这些语法归纳偏置方法在低资源环境中产生不均匀的结果，并在大多数情况下提供出乎意料的少量益处。

Transformer 语言模型中的句法归纳偏置：对低资源语言特别有帮助吗？

Syntactic Inductive Bias in Transformer Language Models: Especially  Helpful for Low-Resource Languages?

The dominant language modeling paradigm handles text as a sequence of
discrete tokens. While that approach can capture the latent structure of the
text, it is inherently constrained to sequential dynamics for text generation.
We propose a new paradigm for introducing a syntactic inductive bias into
neural text generation, where the dependency parse tree is used to drive the
Transformer model to generate sentences iteratively.
Our experiments show that this paradigm is effective at text generation, with
quality between LSTMs and Transformers, and comparable diversity, requiring
less than half their decoding steps, and its generation process allows direct
control over the syntactic constructions of the generated text, enabling the
induction of stylistic variations.

通过引入句法归纳偏差，使用依赖分析树驱动转换器模型迭代生成句子，实验证明其比 LSTM 和 Transformer 更有效，并具有可比拟的多样性且需要较少的解码步骤，生成过程允许对生成文本的句法构造进行直接控制，从而实现风格变形的归纳。