In this study, we present an investigation into the anisotropy dynamics and
intrinsic dimension of embeddings in transformer architectures, focusing on the
dichotomy between encoders and decoders. Our findings reveal that the
anisotropy profile in transformer decoders exhibits a distinct bell-shaped
curve, with the highest anisotropy concentrations in the middle layers. This
pattern diverges from the more uniformly distributed anisotropy observed in
encoders. In addition, we found that the intrinsic dimension of embeddings
increases in the initial phases of training, indicating an expansion into
higher-dimensional space. Which is then followed by a compression phase towards
the end of training with dimensionality decrease, suggesting a refinement into
more compact representations. Our results provide fresh insights to the
understanding of encoders and decoders embedding properties.

我们研究了 Transformer 结构中编码器和解码器之间的各向异性动态和内部维度，并发现解码器中的各向异性特征呈现出一个具有峰型曲线的独特模式，峰值位于中间层，与编码器中更均匀分布的各向异性特征不同。此外，我们发现嵌入的内部维度在训练的初始阶段增加，表明向更高维空间扩展，然后在训练末期经历降维的压缩阶段，暗示更紧凑的表示的提炼过程。我们的研究结果为理解编码器和解码器的嵌入特性提供了新的见解。