Transformer-based models have recently become wildly successful across a diverse set of domains. At the same time, recent work has shown that Transformers are inherently low-pass filters that gradually oversmooth the inputs, reducing the expressivity of their representations. A natural question is: How can Transformers achieve these successes given this shortcoming? In this work we show that in fact Transformers are not inherently low-pass filters. Instead, whether Transformers oversmooth or not depends on the eigenspectrum of their update equations. Our analysis extends prior work in oversmoothing and in the closely-related phenomenon of rank collapse. We show that many successful Transformer models have attention and weights which satisfy conditions that avoid oversmoothing. Based on this analysis, we derive a simple way to parameterize the weights of the Transformer update equations that allows for control over its spectrum, ensuring that oversmoothing does not occur. Compared to a recent solution for oversmoothing, our approach improves generalization, even when training with more layers, fewer datapoints, and data that is corrupted.

Transformer模型在不同领域取得了巨大的成功，但近期的研究发现，Transformer本质上是低通滤波器，会逐渐过度平滑输入，从而降低其表达能力。然而，根据本研究，Transformer是否过度平滑取决于其更新方程的特征频谱，因此提出了一种简洁的参数化方法，能够控制Transformer的频谱，避免过度平滑。相比于近期的解决方法，该方法提高了泛化性能，即使在更多层次、更少数据点和受损数据的训练情况下仍然有效。

Transformer过度平滑的真相