A line of work on Transformer-based language models such as BERT has
attempted to use syntactic inductive bias to enhance the pretraining process,
on the theory that building syntactic structure into the training process
should reduce the amount of data needed for training. But such methods are
often tested for high-resource languages such as English. In this work, we
investigate whether these methods can compensate for data sparseness in
low-resource languages, hypothesizing that they ought to be more effective for
low-resource languages. We experiment with five low-resource languages: Uyghur,
Wolof, Maltese, Coptic, and Ancient Greek. We find that these syntactic
inductive bias methods produce uneven results in low-resource settings, and
provide surprisingly little benefit in most cases.

一项关于基于 Transformer 的语言模型（如 BERT）的研究尝试使用语法归纳偏置来增强预训练过程，理论上通过将语法结构融入训练过程可以降低训练所需的数据量。但此类方法通常在高资源语言（如英语）上进行测试。在这项研究中，我们调查了这些方法是否可以弥补低资源语言中的数据稀疏性，研究假设它们在低资源语言中应该更加有效。我们对五种低资源语言进行了实验：维吾尔语、沃洛夫语、马耳他语、科普特语和古希腊语。我们发现这些语法归纳偏置方法在低资源环境中产生不均匀的结果，并在大多数情况下提供出乎意料的少量益处。

Transformer 语言模型中的句法归纳偏置：对低资源语言特别有帮助吗？

Syntactic Inductive Bias in Transformer Language Models: Especially  Helpful for Low-Resource Languages?

Neural machine translation (NMT) has achieved impressive performance on
machine translation task in recent years. However, in consideration of
efficiency, a limited-size vocabulary that only contains the top-N highest
frequency words are employed for model training, which leads to many rare and
unknown words. It is rather difficult when translating from the low-resource
and morphologically-rich agglutinative languages, which have complex morphology
and large vocabulary. In this paper, we propose a morphological word
segmentation method on the source-side for NMT that incorporates morphology
knowledge to preserve the linguistic and semantic information in the word
structure while reducing the vocabulary size at training time. It can be
utilized as a preprocessing tool to segment the words in agglutinative
languages for other natural language processing (NLP) tasks. Experimental
results show that our morphologically motivated word segmentation method is
better suitable for the NMT model, which achieves significant improvements on
Turkish-English and Uyghur-Chinese machine translation tasks on account of
reducing data sparseness and language complexity.

本研究提出了一种基于形态学的单词分割方法，能够在训练时缩小词汇表的同时保留词结构中的语言和语义信息，以协助神经机器翻译，实验结果表明此方法能显著提高土耳其 - 英语和维吾尔 - 中文机器翻译任务的性能，降低数据稀疏性和语言复杂性。

神经机器翻译上的聚合语言词形分割

Morphological Word Segmentation on Agglutinative Languages for Neural  Machine Translation

Capturing the semantic relations of words in a vector space contributes to
many natural language processing tasks. One promising approach exploits
lexico-syntactic patterns as features of word pairs. In this paper, we propose
a novel model of this pattern-based approach, neural latent relational analysis
(NLRA). NLRA can generalize co-occurrences of word pairs and lexico-syntactic
patterns, and obtain embeddings of the word pairs that do not co-occur. This
overcomes the critical data sparseness problem encountered in previous
pattern-based models. Our experimental results on measuring relational
similarity demonstrate that NLRA outperforms the previous pattern-based models.
In addition, when combined with a vector offset model, NLRA achieves a
performance comparable to that of the state-of-the-art model that exploits
additional semantic relational data.

本文提出了一种基于模式的新模型，神经潜在关系分析（NLRA），旨在捕捉向量空间中词汇关系的语义，可以泛化词对和词汇 - 语法模式的共现，同时解决了传统基于模式的模型中遇到的严重数据稀疏性问题，实验结果表明 NLRA 在测量关系相似度方面的表现优于先前的基于模式的模型，同时与矢量偏移模型相结合时可以达到与利用其他语义关系数据的现有模型相当的性能。