Transformers-based pretrained language models achieve outstanding results in
many well-known NLU benchmarks. However, while pretraining methods are very
convenient, they are expensive in terms of time and resources. This calls for a
study of the impact of pretraining data size on the knowledge of the models. We
explore this impact on the syntactic capabilities of RoBERTa, using models
trained on incremental sizes of raw text data. First, we use syntactic
structural probes to determine whether models pretrained on more data encode a
higher amount of syntactic information. Second, we perform a targeted syntactic
evaluation to analyze the impact of pretraining data size on the syntactic
generalization performance of the models. Third, we compare the performance of
the different models on three downstream applications: part-of-speech tagging,
dependency parsing and paraphrase identification. We complement our study with
an analysis of the cost-benefit trade-off of training such models. Our
experiments show that while models pretrained on more data encode more
syntactic knowledge and perform better on downstream applications, they do not
always offer a better performance across the different syntactic phenomena and
come at a higher financial and environmental cost.

本研究探讨了预训练数据大小对 RoBERTa 模型的句法能力及其在下游应用中的影响，并分析了训练此类模型的成本效益权衡。结果显示，虽然预训练数据大小的增加会显著提高模型的句法能力及在下游任务中表现，但这也带来了更高的经济和环境成本。