Although models using contextual word embeddings have achieved state-of-the-art results on a host of NLP tasks, little is known about exactly what information these embeddings encode about the context words that they are understood to reflect. To address this question, we introduce a suite of probing tasks that enable fine-grained testing of contextual embeddings for encoding of information about surrounding words. We apply these tasks to examine the popular BERT, ELMo and GPT contextual encoders, and find that each of our tested information types is indeed encoded as contextual information across tokens, often with near-perfect recoverability-but the encoders vary in which features they distribute to which tokens, how nuanced their distributions are, and how robust the encoding of each feature is to distance. We discuss implications of these results for how different types of models breakdown and prioritize word-level context information when constructing token embeddings.

通过引入一系列探测任务，对比测试了BERT、ELMo和GPT等语境编码器对于上下文信息的编码能力，发现每种信息类型确实都是以上下文信息的形式编码的，但是不同的编码器在分配特征到哪些标记上、特征分配的细度、特征编码的鲁棒性等方面存在差异。

窥探邻居：针对上下文嵌入的细粒度探测，用于获取周围词语的信息