Large pretrained language models (PLMs) typically tokenize the input string
into contiguous subwords before any pretraining or inference. However, previous
studies have claimed that this form of subword tokenization is inadequate for
processing morphologically-rich languages (MRLs). We revisit this hypothesis by
pretraining a BERT-style masked language model over character sequences instead
of word-pieces. We compare the resulting model, dubbed TavBERT, against
contemporary PLMs based on subwords for three highly complex and ambiguous MRLs
(Hebrew, Turkish, and Arabic), testing them on both morphological and semantic
tasks. Our results show, for all tested languages, that while TavBERT obtains
mild improvements on surface-level tasks à la POS tagging and full
morphological disambiguation, subword-based PLMs achieve significantly higher
performance on semantic tasks, such as named entity recognition and extractive
question answering. These results showcase and (re)confirm the potential of
subword tokenization as a reasonable modeling assumption for many languages,
including MRLs.

对于语言中包含丰富词形的复杂语言，使用预训练字符序列的 BERT-style 掩码语言模型进行训练和推理而不是基于子单元的方式可能会得到更好的表现。但是，对于语义任务，基于子单元的 PLM 表现更好，这证实了基于子单元的分词作为许多语言的合理建模假设的潜力。