A recent body of work has demonstrated that Transformer embeddings can be
linearly decomposed into well-defined sums of factors, that can in turn be
related to specific network inputs or components. There is however still a
dearth of work studying whether these mathematical reformulations are
empirically meaningful. In the present work, we study representations from
machine-translation decoders using two of such embedding decomposition methods.
Our results indicate that, while decomposition-derived indicators effectively
correlate with model performance, variation across different runs suggests a
more nuanced take on this question. The high variability of our measurements
indicate that geometry reflects model-specific characteristics more than it
does sentence-specific computations, and that similar training conditions do
not guarantee similar vector spaces.

通过两种嵌入分解方法，研究机器翻译解码器的表示，结果表明分解导出的指标与模型性能有效相关，但不同运行之间的变化暗示了对这个问题更微妙的看法，测量结果的高变异性表明几何更多地反映模型特定的特征而不是句子特定的计算，并且类似的训练条件不能保证相似的向量空间。