In recent years there have been considerable advances in pre-trained language
models, where non-English language versions have also been made available. Due
to their increasing use, many lightweight versions of these models (with
reduced parameters) have also been released to speed up training and inference
times. However, versions of these lighter models (e.g., ALBERT, DistilBERT) for
languages other than English are still scarce. In this paper we present ALBETO
and DistilBETO, which are versions of ALBERT and DistilBERT pre-trained
exclusively on Spanish corpora. We train several versions of ALBETO ranging
from 5M to 223M parameters and one of DistilBETO with 67M parameters. We
evaluate our models in the GLUES benchmark that includes various natural
language understanding tasks in Spanish. The results show that our lightweight
models achieve competitive results to those of BETO (Spanish-BERT) despite
having fewer parameters. More specifically, our larger ALBETO model outperforms
all other models on the MLDoc, PAWS-X, XNLI, MLQA, SQAC and XQuAD datasets.
However, BETO remains unbeaten for POS and NER. As a further contribution, all
models are publicly available to the community for future research.

在本文中，我们针对西班牙语语料库训练了两种语言模型 ALBETO 和 DistilBETO，这些模型参数轻量级，且相较于其他模型在西班牙语自然语言解析任务上取得了竞争性的结果。