Contextual word representations derived from large-scale neural language
models are successful across a diverse set of NLP tasks, suggesting that they
encode useful and transferable features of language. To shed light on the
linguistic knowledge they capture, we study the representations produced by
several recent pretrained contextualizers (variants of ELMo, the OpenAI
transformer language model, and BERT) with a suite of seventeen diverse probing
tasks. We find that linear models trained on top of frozen contextual
representations are competitive with state-of-the-art task-specific models in
many cases, but fail on tasks requiring fine-grained linguistic knowledge
(e.g., conjunct identification). To investigate the transferability of
contextual word representations, we quantify differences in the transferability
of individual layers within contextualizers, especially between recurrent
neural networks (RNNs) and transformers. For instance, higher layers of RNNs
are more task-specific, while transformer layers do not exhibit the same
monotonic trend. In addition, to better understand what makes contextual word
representations transferable, we compare language model pretraining with eleven
supervised pretraining tasks. For any given task, pretraining on a closely
related task yields better performance than language model pretraining (which
is better on average) when the pretraining dataset is fixed. However, language
model pretraining on more data gives the best results.

该论文研究了利用大规模神经语言模型生成的上下文词表示对于自然语言处理任务的有效性及其可迁移性。结果表明，虽然这些表示在许多任务中表现出色，但对于需要细粒度语言知识的任务（如连词识别）而言，它们还不能胜任。此外，作者还比较了不同预训练和监督预训练方法对于任务训练的影响。