Pre-trained language models (PLMs) have shown remarkable successes in
acquiring a wide range of linguistic knowledge, relying solely on
self-supervised training on text streams. Nevertheless, the effectiveness of
this language-agnostic approach has been frequently questioned for its
sub-optimal performance when applied to morphologically-rich languages (MRLs).
We investigate the hypothesis that incorporating explicit morphological
knowledge in the pre-training phase can improve the performance of PLMs for
MRLs. We propose various morphologically driven tokenization methods enabling
the model to leverage morphological cues beyond raw text. We pre-train multiple
language models utilizing the different methods and evaluate them on Hebrew, a
language with complex and highly ambiguous morphology. Our experiments show
that morphologically driven tokenization demonstrates improved results compared
to a standard language-agnostic tokenization, on a benchmark of both semantic
and morphologic tasks. These findings suggest that incorporating morphological
knowledge holds the potential for further improving PLMs for morphologically
rich languages.

将显式的形态学知识引入预训练阶段可以改善用于形态学丰富语言的预训练语言模型的性能，并且基于形态学驱动的分词方法相比普通的通用语言分词方法在语义和形态任务的基准测试中显示出了改进的结果，这些发现表明将形态学知识整合进预训练语言模型中具有进一步提高形态学丰富语言性能的潜力。