This paper presents ViDeBERTa, a new pre-trained monolingual language model
for Vietnamese, with three versions - ViDeBERTa_xsmall, ViDeBERTa_base, and
ViDeBERTa_large, which are pre-trained on a large-scale corpus of high-quality
and diverse Vietnamese texts using DeBERTa architecture. Although many
successful pre-trained language models based on Transformer have been widely
proposed for the English language, there are still few pre-trained models for
Vietnamese, a low-resource language, that perform good results on downstream
tasks, especially Question answering. We fine-tune and evaluate our model on
three important natural language downstream tasks, Part-of-speech tagging,
Named-entity recognition, and Question answering. The empirical results
demonstrate that ViDeBERTa with far fewer parameters surpasses the previous
state-of-the-art models on multiple Vietnamese-specific natural language
understanding tasks. Notably, ViDeBERTa_base with 86M parameters, which is only
about 23% of PhoBERT_large with 370M parameters, still performs the same or
better results than the previous state-of-the-art model. Our ViDeBERTa models
are available at: this https URL.

本文介绍了 ViDeBERTa，一种新的越南语预训练单语言模型，使用 DeBERTa 架构在大规模高质量多样化的越南文本语料库上进行预训练，取得了在越南特定的自然语言理解任务上比前文献更好的表现，尤其在问答任务上能取得出色的结果。该模型相较于其他预训练语言模型具有更少的参数，其表现非常出色，并且现已提供供大家使用。