Recent advances in NLP have significantly improved the performance of
language models on a variety of tasks. While these advances are largely driven
by the availability of large amounts of data and computational power, they also
benefit from the development of better training methods and architectures. In
this paper, we introduce CamemBERTa, a French DeBERTa model that builds upon
the DeBERTaV3 architecture and training objective. We evaluate our model's
performance on a variety of French downstream tasks and datasets, including
question answering, part-of-speech tagging, dependency parsing, named entity
recognition, and the FLUE benchmark, and compare against CamemBERT, the
state-of-the-art monolingual model for French. Our results show that, given the
same amount of training tokens, our model outperforms BERT-based models trained
with MLM on most tasks. Furthermore, our new model reaches similar or superior
performance on downstream tasks compared to CamemBERT, despite being trained on
only 30% of its total number of input tokens. In addition to our experimental
results, we also publicly release the weights and code implementation of
CamemBERTa, making it the first publicly available DeBERTaV3 model outside of
the original paper and the first openly available implementation of a DeBERTaV3
training objective. this https URL

介绍了一种基于 DeBERTaV3 架构和训练目标的法语 DeBERTa 模型 CamemBERTa，通过在各种法语下游任务和数据集上的性能评估，证明了该模型在训练令牌相同的情况下优于大多数基于 BERT 的模型，并且在性能上与 CamemBERT 类似或更好，而只用了后者总输入令牌数量的 30％。同时，也公开了该模型的权重和代码实现，是 DeBERTaV3 模型的第一个公开实现及应用。