Large pretrained language models have recently conquered the area of natural
language processing. As an alternative to predominant masked language modelling
introduced in BERT, the T5 model has introduced a more general training
objective, namely sequence to sequence transformation, which includes masked
language model but more naturally fits text generation tasks such as machine
translation, summarization, question answering, text simplification, dialogue
systems, etc. The monolingual variants of T5 models have been limited to
well-resourced languages, while the massively multilingual T5 model supports
101 languages. In contrast, we trained two different sized T5-type sequence to
sequence models for morphologically rich Slovene language with much less
resources and analyzed their behavior on 11 tasks. Concerning classification
tasks, the SloT5 models mostly lag behind the monolingual Slovene SloBERTa
model but are useful for the generative tasks.

该研究针对资源较少的斯洛文尼亚语，训练了两个不同尺寸的 T5 类型序列到序列模型，并分析其在 11 个任务中的表现，结果显示 SloT5 模型在分类任务上大多落后于单语斯洛文尼亚 SloBERTa 模型，但在生成性任务上很有用。