Semitic morphologically-rich languages (MRLs) are characterized by extreme
word ambiguity. Because most vowels are omitted in standard texts, many of the
words are homographs with multiple possible analyses, each with a different
pronunciation and different morphosyntactic properties. This ambiguity goes
beyond word-sense disambiguation (WSD), and may include token segmentation into
multiple word units. Previous research on MRLs claimed that standardly trained
pre-trained language models (PLMs) based on word-pieces may not sufficiently
capture the internal structure of such tokens in order to distinguish between
these analyses. Taking Hebrew as a case study, we investigate the extent to
which Hebrew homographs can be disambiguated and analyzed using PLMs. We
evaluate all existing models for contextualized Hebrew embeddings on a novel
Hebrew homograph challenge sets that we deliver. Our empirical results
demonstrate that contemporary Hebrew contextualized embeddings outperform
non-contextualized embeddings; and that they are most effective for
disambiguating segmentation and morphosyntactic features, less so regarding
pure word-sense disambiguation. We show that these embeddings are more
effective when the number of word-piece splits is limited, and they are more
effective for 2-way and 3-way ambiguities than for 4-way ambiguity. We show
that the embeddings are equally effective for homographs of both balanced and
skewed distributions, whether calculated as masked or unmasked tokens. Finally,
we show that these embeddings are as effective for homograph disambiguation
with extensive supervised training as with a few-shot setup.

研究表明，最新的希伯来语情境嵌入模型在消除希伯来语同形异义词和分析词素、句法特征方面效果显著，尤其在限制分词数、2 种和 3 种歧义的情况下，对于平衡和不平衡分布的同形异义词同样有效，无论是有监督训练还是几次训练。

预训练上下文语言模型对希伯来同形字分析的区分能力

Do Pretrained Contextual Language Models Distinguish between Hebrew  Homograph Analyses?

We present a new pre-trained language model (PLM) for Rabbinic Hebrew, termed
Berel (BERT Embeddings for Rabbinic-Encoded Language). Whilst other PLMs exist
for processing Hebrew texts (e.g., HeBERT, AlephBert), they are all trained on
modern Hebrew texts, which diverges substantially from Rabbinic Hebrew in terms
of its lexicographical, morphological, syntactic and orthographic norms. We
demonstrate the superiority of Berel on Rabbinic texts via a challenge set of
Hebrew homographs. We release the new model and homograph challenge set for
unrestricted use.

本文介绍了一种新的面向拉比希伯来语的预训练语言模型 Berel，该模型在处理拉比希伯来语言文本方面的效果优于现有的模型。作者通过希伯来同形异义词的挑战集证明了 Berel 的优越性，并将其模型和同形异义词挑战集公开提供给任何人免费使用。