Autoregressive language models, pretrained using large text corpora to do
well on next word prediction, have been successful at solving many downstream
tasks, even with zero-shot usage. However, there is little theoretical
understanding of this success. This paper initiates a mathematical study of
this phenomenon for the downstream task of text classification by considering
the following questions: (1) What is the intuitive connection between the
pretraining task of next word prediction and text classification? (2) How can
we mathematically formalize this connection and quantify the benefit of
language modeling? For (1), we hypothesize, and verify empirically, that
classification tasks of interest can be reformulated as sentence completion
tasks, thus making language modeling a meaningful pretraining task. With a
mathematical formalization of this hypothesis, we make progress towards (2) and
show that language models that are $\epsilon$-optimal in cross-entropy
(log-perplexity) learn features that can linearly solve such classification
tasks with $\mathcal{O}(\sqrt{\epsilon})$ error, thus demonstrating that doing
well on language modeling can be beneficial for downstream tasks. We
experimentally verify various assumptions and theoretical findings, and also
use insights from the analysis to design a new objective function that performs
well on some classification tasks.

本文通过数学研究自回归语言模型预训练在下游任务中的应用，提出了将分类任务转化为句子填充任务的假设，证实表现好的语言模型可以是有意义的预训练任务，并给出了相应的数学形式化，同时通过分析认为语言模型可以有助于线性解决分类任务。