Since the appearance of BERT, recent works including XLNet and RoBERTa
utilize sentence embedding models pre-trained by large corpora and a large
number of parameters. Because such models have large hardware and a huge amount
of data, they take a long time to pre-train. Therefore it is important to
attempt to make smaller models that perform comparatively. In this paper, we
trained a Korean-specific model KR-BERT, utilizing a smaller vocabulary and
dataset. Since Korean is one of the morphologically rich languages with poor
resources using non-Latin alphabets, it is also important to capture
language-specific linguistic phenomena that the Multilingual BERT model missed.
We tested several tokenizers including our BidirectionalWordPiece Tokenizer and
adjusted the minimal span of tokens for tokenization ranging from sub-character
level to character-level to construct a better vocabulary for our model. With
those adjustments, our KR-BERT model performed comparably and even better than
other existing pre-trained models using a corpus about 1/10 of the size.

本论文旨在训练适用于韩语的小型语言模型 KR-BERT，通过采用较小的词汇表和数据集、优化令牌化方法，提高了韩语语言现象的捕捉效率，实现了与大型语言模型相媲美甚至更优的性能。