We introduce a framework for analyzing various types of information in an NLP
Transformer. In this approach, we distinguish four layers of information:
positional, syntactic, semantic, and contextual. We also argue that the common
practice of adding positional information to semantic embedding is sub-optimal
and propose instead a Linear-and-Add approach. Our analysis reveals an
autogenetic separation of positional information through the deep layers. We
show that the distilled positional components of the embedding vectors follow
the path of a helix, both on the encoder side and on the decoder side. We
additionally show that on the encoder side, the conceptual dimensions generate
Part-of-Speech (PoS) clusters. On the decoder side, we show that a di-gram
approach helps to reveal the PoS clusters of the next token. Our approach paves
a way to elucidate the processing of information through the deep layers of an
NLP Transformer.

本研究提出了通过分析 NLP Transformer 中的不同信息类型的框架，并通过 Linear-and-Add 方法优化位置信息嵌入，发现位置信息具有深度分离特征，进而证明其对于 encoder 处的词性聚类和 decoder 处的下一个 token 的词性聚类具有启发意义。