S\'ami, an indigenous language group comprising multiple languages, faces
digital marginalization due to the limited availability of data and
sophisticated language models designed for its linguistic intricacies. This
work focuses on increasing technological participation for the S\'ami language.
We draw the attention of the ML community towards the language modeling problem
of Ultra Low Resource (ULR) languages. ULR languages are those for which the
amount of available textual resources is very low, and the speaker count for
them is also very low. ULRLs are also not supported by mainstream Large
Language Models (LLMs) like ChatGPT, due to which gathering artificial training
data for them becomes even more challenging. Mainstream AI foundational model
development has given less attention to this category of languages. Generally,
these languages have very few speakers, making it hard to find them. However,
it is important to develop foundational models for these ULR languages to
promote inclusion and the tangible abilities and impact of LLMs. To this end,
we have compiled the available S\'ami language resources from the web to create
a clean dataset for training language models. In order to study the behavior of
modern LLM models with ULR languages (S\'ami), we have experimented with
different kinds of LLMs, mainly at the order of $\sim$ seven billion
parameters. We have also explored the effect of multilingual LLM training for
ULRLs. We found that the decoder-only models under a sequential multilingual
training scenario perform better than joint multilingual training, whereas
multilingual training with high semantic overlap, in general, performs better
than training from scratch.This is the first study on the S\'ami language for
adapting non-statistical language models that use the latest developments in
the field of natural language processing (NLP).

该研究致力于解决 S'ami 语言面临的数字边缘化问题，通过采用最新的自然语言处理技术，训练基于超低资源语言的语言模型，并通过多语言训练实验，发现在顺序多语言训练情景下，解码器模型的表现优于联合多语言训练，且高语义重叠的多语言训练比从头训练要更好。