The success of pretrained contextual encoders, such as ELMo and BERT, has
brought a great deal of interest in what these models learn: do they, without
explicit supervision, learn to encode meaningful notions of linguistic
structure? If so, how is this structure encoded? To investigate this, we
introduce latent subclass learning (LSL): a modification to existing
classifier-based probing methods that induces a latent categorization (or
ontology) of the probe's inputs. Without access to fine-grained gold labels,
LSL extracts emergent structure from input representations in an interpretable
and quantifiable form. In experiments, we find strong evidence of familiar
categories, such as a notion of personhood in ELMo, as well as novel
ontological distinctions, such as a preference for fine-grained semantic roles
on core arguments. Our results provide unique new evidence of emergent
structure in pretrained encoders, including departures from existing
annotations which are inaccessible to earlier methods.

本研究介绍了一种名为潜在子类学习的方法，通过这种方法可以从输入表示中提取出诸如人的概念等熟悉的类别，以及对于核心参数的细粒度语义角色的首选，这些结果为预训练编码器中的新兴结构提供了独特的新证据。