We train several language models for Icelandic, including IceBERT, that
achieve state-of-the-art performance in a variety of downstream tasks,
including part-of-speech tagging, named entity recognition, grammatical error
detection and constituency parsing. To train the models we introduce a new
corpus of Icelandic text, the Icelandic Common Crawl Corpus (IC3), a collection
of high quality texts found online by targeting the Icelandic top-level-domain
(TLD). Several other public data sources are also collected for a total of 16GB
of Icelandic text. To enhance the evaluation of model performance and to raise
the bar in baselines for Icelandic, we translate and adapt the WinoGrande
dataset for co-reference resolution. Through these efforts we demonstrate that
a properly cleaned crawled corpus is sufficient to achieve state-of-the-art
results in NLP applications for low to medium resource languages, by comparison
with models trained on a curated corpus. We further show that initializing
models using existing multilingual models can lead to state-of-the-art results
for some downstream tasks.

本文介绍了利用冰岛语通用爬虫语料库进行训练的几种语言模型，包括 IceBERT，这些模型在诸多下游任务中均达到了最优表现。通过这些努力，我们证明了经过适当清理的网络爬虫语料库足以实现自然语言处理应用的最佳效果，并且说明使用现有的多语言模型初始化可以在某些下游任务中达到最先进的结果。