This paper presents an approach for adapting the DebertaV3 XSmall model
pre-trained in English for Brazilian Portuguese natural language processing
(NLP) tasks. A key aspect of the methodology involves a multistep training
process to ensure the model is effectively tuned for the Portuguese language.
Initial datasets from Carolina and BrWac are preprocessed to address issues
like emojis, HTML tags, and encodings. A Portuguese-specific vocabulary of
50,000 tokens is created using SentencePiece. Rather than training from
scratch, the weights of the pre-trained English model are used to initialize
most of the network, with random embeddings, recognizing the expensive cost of
training from scratch. The model is fine-tuned using the replaced token
detection task in the same format of DebertaV3 training. The adapted model,
called DeBERTinha, demonstrates effectiveness on downstream tasks like named
entity recognition, sentiment analysis, and determining sentence relatedness,
outperforming BERTimbau-Large in two tasks despite having only 40M parameters.

这篇论文介绍了一种将英语预训练的 DebertaV3 XSmall 模型适应于巴西葡萄牙语自然语言处理任务的方法。该方法的关键是多步骤训练过程，以确保模型在葡萄牙语方面得到有效调整。通过对来自 Carolina 和 BrWac 的初始数据集进行预处理，解决了表情符号、HTML 标签和编码等问题。使用 SentencePiece 创建了一个包含 50,000 个令牌的葡萄牙语特定词汇表。模型使用预训练的英语模型的权重初始化大部分网络，通过随机嵌入来减少从头训练的昂贵成本。通过在 DebertaV3 训练的相同格式中使用替换令牌检测任务对模型进行微调。改进后的模型称为 DeBERTinha，在命名实体识别、情感分析和句子相关性确定等下游任务中表现出色，在两个任务中超过 BERTimbau-Large，尽管其参数只有 40M。