We present IndoBERTweet, the first large-scale pretrained model for
Indonesian Twitter that is trained by extending a monolingually-trained
Indonesian BERT model with additive domain-specific vocabulary. We focus in
particular on efficient model adaptation under vocabulary mismatch, and
benchmark different ways of initializing the BERT embedding layer for new word
types. We find that initializing with the average BERT subword embedding makes
pretraining five times faster, and is more effective than proposed methods for
vocabulary adaptation in terms of extrinsic evaluation over seven Twitter-based
datasets.

本研究介绍了 IndoBERTweet，这是第一个大规模预训练的印度尼西亚语 Twitter 模型，它通过扩展单语训练的印度尼西亚 BERT 模型来训练，并使用了特定领域的词汇表。我们特别关注词汇不匹配下的有效模型适应性，并基准测试了不同的 BERT 嵌入层初始化方式，发现用平均 BERT 子词嵌入初始化可以使预训练速度提高五倍，并且在七个基于 Twitter 的数据集上使用外部评估比提出的词汇自适应方法更有效。