In this work, we study the representation space of contextualized embeddings and gain insight into the hidden topology of large language models. We show there exists a network of latent states that summarize linguistic properties of contextualized representations. Instead of seeking alignments to existing well-defined annotations, we infer this latent network in a fully unsupervised way using a structured variational autoencoder. The induced states not only serve as anchors that mark the topology (neighbors and connectivity) of the representation manifold but also reveal the internal mechanism of encoding sentences. With the induced network, we: (1). decompose the representation space into a spectrum of latent states which encode fine-grained word meanings with lexical, morphological, syntactic and semantic information; (2). show state-state transitions encode rich phrase constructions and serve as the backbones of the latent space. Putting the two together, we show that sentences are represented as a traversal over the latent network where state-state transition chains encode syntactic templates and state-word emissions fill in the content. We demonstrate these insights with extensive experiments and visualizations.

通过使用结构化的变分自编码器进行完全无监督的学习，得出语境化嵌入的表征空间存在潜在状态的网络，这些状态不仅作为表示流形的拓扑结构的锚点，还揭示了对句子进行编码的内部机制。同时，表明句子作为潜在网络上的遍历，状态转换链编码了句法模板，状态-词发射填充了内容。

理解语境化表示的潜在拓扑感知