NLP is currently dominated by general-purpose pretrained language models like
RoBERTa, which achieve strong performance on NLU tasks through pretraining on
billions of words. But what exact knowledge or skills do Transformer LMs learn
from large-scale pretraining that they cannot learn from less data? We adopt
four probing methods---classifier probing, information-theoretic probing,
unsupervised relative acceptability judgment, and fine-tuning on NLU
tasks---and draw learning curves that track the growth of these different
measures of linguistic ability with respect to pretraining data volume using
the MiniBERTas, a group of RoBERTa models pretrained on 1M, 10M, 100M and 1B
words. We find that LMs require only about 10M or 100M words to learn
representations that reliably encode most syntactic and semantic features we
test. A much larger quantity of data is needed in order to acquire enough
commonsense knowledge and other skills required to master typical downstream
NLU tasks. The results suggest that, while the ability to encode linguistic
features is almost certainly necessary for language understanding, it is likely
that other forms of knowledge are the major drivers of recent improvements in
language understanding among large pretrained models.

通过四种探究方法，我们发现语言模型只需要大约 100M 的单词量，就能够可靠地编码大多数句法和语义特征，而大量的数据需要用来获得足够的常识和其他技能，以掌握典型的下游 NLU 任务。