Large transformer-based language models, e.g. BERT and GPT-3, outperform
previous architectures on most natural language processing tasks. Such language
models are first pre-trained on gigantic corpora of text and later used as
base-model for finetuning on a particular task. Since the pre-training step is
usually not repeated, base models are not up-to-date with the latest
information. In this paper, we update RobBERT, a RoBERTa-based state-of-the-art
Dutch language model, which was trained in 2019. First, the tokenizer of
RobBERT is updated to include new high-frequent tokens present in the latest
Dutch OSCAR corpus, e.g. corona-related words. Then we further pre-train the
RobBERT model using this dataset. To evaluate if our new model is a plug-in
replacement for RobBERT, we introduce two additional criteria based on concept
drift of existing tokens and alignment for novel tokens.We found that for
certain language tasks this update results in a significant performance
increase. These results highlight the benefit of continually updating a
language model to account for evolving language use.

该研究对 RobBERT 进行了更新，更新了 tokenizer 和数据集，使用新模型取代模型更新的效果显著，证明持续更新语言模型可以提高性能。

RobBERT-2022: 更新荷兰语言模型以适应语言使用的演变

RobBERT-2022: Updating a Dutch Language Model to Account for Evolving Language Use

Pre-trained large-scale language models such as BERT have gained a lot of
attention thanks to their outstanding performance on a wide range of natural
language tasks. However, due to their large number of parameters, they are
resource-intensive both to deploy and to fine-tune. Researchers have created
several methods for distilling language models into smaller ones to increase
efficiency, with a small performance trade-off. In this paper, we create
several different distilled versions of the state-of-the-art Dutch RobBERT
model and call them RobBERTje. The distillations differ in their distillation
corpus, namely whether or not they are shuffled and whether they are merged
with subsequent sentences. We found that the performance of the models using
the shuffled versus non-shuffled datasets is similar for most tasks and that
randomly merging subsequent sentences in a corpus creates models that train
faster and perform better on tasks with long sequences. Upon comparing
distillation architectures, we found that the larger DistilBERT architecture
worked significantly better than the Bort hyperparametrization. Interestingly,
we also found that the distilled models exhibit less gender-stereotypical bias
than its teacher model. Since smaller architectures decrease the time to
fine-tune, these models allow for more efficient training and more lightweight
deployment of many Dutch downstream language tasks.

该论文讨论预训练模型以及使用语言模型精简方法创建的 RobBERTje 蒸馏模型，发现在包含长序列的任务上，与其教师模型相比，蒸馏模型具有更少的性别刻板印象和更好的训练和性能效果。

RobBERTje: 一个被蒸馏的荷兰 BERT 模型

RobBERTje: a Distilled Dutch BERT Model

Pre-trained language models have been dominating the field of natural
language processing in recent years, and have led to significant performance
gains for various complex natural language tasks. One of the most prominent
pre-trained language models is BERT, which was released as an English as well
as a multilingual version. Although multilingual BERT performs well on many
tasks, recent studies show that BERT models trained on a single language
significantly outperform the multilingual version. Training a Dutch BERT model
thus has a lot of potential for a wide range of Dutch NLP tasks. While previous
approaches have used earlier implementations of BERT to train a Dutch version
of BERT, we used RoBERTa, a robustly optimized BERT approach, to train a Dutch
language model called RobBERT. We measured its performance on various tasks as
well as the importance of the fine-tuning dataset size. We also evaluated the
importance of language-specific tokenizers and the model's fairness. We found
that RobBERT improves state-of-the-art results for various tasks, and
especially significantly outperforms other models when dealing with smaller
datasets. These results indicate that it is a powerful pre-trained model for a
large variety of Dutch language tasks. The pre-trained and fine-tuned models
are publicly available to support further downstream Dutch NLP applications.

本研究使用 RobBERT 作为荷兰语预训练模型，对各种任务的执行效果进行了测量，包括用于微调的数据集规模的重要性以及模型的公平性。研究发现，RobBERT 在各种任务上都可以提供良好的性能，并且在处理较小的数据集时明显优于其他模型，这表明它是适用于多种荷兰语任务的功能强大的预训练模型。