Despite advancements in Natural Language Processing (NLP) and the growing
availability of pretrained models, the English language remains the primary
focus of model development. Continued pretraining on language-specific corpora
provides a practical solution for adapting models to other languages. However,
the impact of different pretraining settings on downstream tasks remains
underexplored. This work introduces $\texttt{ptt5-v2}$, investigating the
continued pretraining of T5 models for Portuguese. We first develop a baseline
set of settings and pretrain models with sizes up to 3B parameters. Finetuning
on three Portuguese downstream tasks (assin2 STS, assin2 RTE, and TweetSentBR)
yields SOTA results on the latter two. We then explore the effects of different
pretraining configurations, including quality filters, optimization strategies,
and multi-epoch pretraining. Perhaps surprisingly, their impact remains subtle
compared to our baseline. We release $\texttt{ptt5-v2}$ pretrained checkpoints
and the finetuned MonoT5 rerankers on HuggingFace at
this https URL
and
this https URL

本文介绍了继续预训练 T5 模型的研究，特别是在葡萄牙语方面的应用及其对下游任务的影响，通过实验和优化配置获得了令人满意的结果。