Recent language models use subwording mechanisms to handle
Out-of-Vocabulary(OOV) words seen during test time and, their generation
capacity is generally measured using perplexity, an intrinsic metric. It is
known that increasing the subword granularity results in a decrease of
perplexity value. However, the study of how subwording affects the
understanding capacity of language models has been very few and only limited to
a handful of languages. To reduce this gap we used 6 different tokenization
schemes to pretrain relatively small language models in Nepali and used the
representations learned to finetune on several downstream tasks. Although
byte-level BPE algorithm has been used in recent models like GPT, RoBERTa we
show that on average they are sub-optimal in comparison to algorithms such as
SentencePiece in finetuning performances for Nepali. Additionally, similar
recent studies have focused on the Bert-based language model. We, however,
pretrain and finetune sequential transformer-based language models.

使用 6 种不同的分词方案预先训练了相对较小的尼泊尔语言模型，并发现相比于字节级 BPE 算法，SentencePiece 等算法在尼泊尔的微调性能上表现更好。此外，我们还预先训练和微调了基于顺序转换器的语言模型。