This paper details the process of developing the first native large
generative language model for the Nordic languages, GPT-SW3. We cover all parts
of the development process, from data collection and processing, training
configuration and instruction finetuning, to evaluation and considerations for
release strategies. We hope that this paper can serve as a guide and reference
for other researchers that undertake the development of large generative models
for smaller languages.

本篇论文介绍了第一个本地大型生成语言模型 GPT-SW3 的开发过程，包括数据收集和处理、训练配置和指令微调、评估以及发布策略的考虑，并希望该论文能为其他研究者在小语种的大型生成模型的开发提供指南和参考。

GPT-SW3：针对北欧语言的自回归语言模型

GPT-SW3: An Autoregressive Language Model for the Nordic Languages

Pre-training Large Language Models (LLMs) require massive amounts of text
data, and the performance of the LLMs typically correlates with the scale and
quality of the datasets. This means that it may be challenging to build LLMs
for smaller languages such as Nordic ones, where the availability of text
corpora is limited. In order to facilitate the development of the LLMS in the
Nordic languages, we curate a high-quality dataset consisting of 1.2TB of text,
in all of the major North Germanic languages (Danish, Icelandic, Norwegian, and
Swedish), as well as some high-quality English data. This paper details our
considerations and processes for collecting, cleaning, and filtering the
dataset.

为了推动在诸如北欧语言这样的小语种中开发 LLMs，我们策划了一个高质量的数据集，其中包含所有主要的北日耳曼语言（丹麦语、冰岛语、挪威语和瑞典语），以及一些高质量的英语数据，并详细介绍了我们的数据收集、清理和过滤的过程。

北欧桩：一个用于语言建模的 1.2TB 北欧数据集

The Nordic Pile: A 1.2TB Nordic Dataset for Language Modeling

The multilingual BERT model is trained on 104 languages and meant to serve as
a universal language model and tool for encoding sentences. We explore how well
the model performs on several languages across several tasks: a diagnostic
classification probing the embeddings for a particular syntactic property, a
cloze task testing the language modelling ability to fill in gaps in a
sentence, and a natural language generation task testing for the ability to
produce coherent text fitting a given context. We find that the currently
available multilingual BERT model is clearly inferior to the monolingual
counterparts, and cannot in many cases serve as a substitute for a well-trained
monolingual model. We find that the English and German models perform well at
generation, whereas the multilingual model is lacking, in particular, for
Nordic languages.

本文探讨了多语言 BERT 模型在语言编码、语法特性、语言生成等任务上的表现，发现该模型性能低于单语言模型，在某些情况下无法取代单语言模型，尤其在北欧语言方面表现不足。