Many attempts have been made in multilingual NLP to ensure that pre-trained
language models, such as mBERT or GPT2 get better and become applicable to
low-resource languages. To achieve multilingualism for pre-trained language
models (PLMs), we need techniques to create word embeddings that capture the
linguistic characteristics of any language. Tokenization is one such technique
because it allows for the words to be split based on characters or subwords,
creating word embeddings that best represent the structure of the language.
Creating such word embeddings is essential to applying PLMs to other languages
where the model was not trained, enabling multilingual NLP. However, most PLMs
use generic tokenization methods like BPE, wordpiece, or unigram which may not
suit specific languages. We hypothesize that tokenization based on syllables
within the input text, which we call syllable tokenization, should facilitate
the development of syllable-aware language models. The syllable-aware language
models make it possible to apply PLMs to languages that are rich in syllables,
for instance, Swahili. Previous works introduced subword tokenization. Our work
extends such efforts. Notably, we propose a syllable tokenizer and adopt an
experiment-centric approach to validate the proposed tokenizer based on the
Swahili language. We conducted text-generation experiments with GPT2 to
evaluate the effectiveness of the syllable tokenizer. Our results show that the
proposed syllable tokenizer generates syllable embeddings that effectively
represent the Swahili language.

我们提出了一种音节分词器，并采用实验为中心的方法验证所提出的分词器在斯瓦希里语中的有效性。我们使用 GPT2 进行文本生成实验，结果证明所提出的音节分词器能够有效地表示斯瓦希里语。