Transformers are extremely successful machine learning models whose mathematical properties remain poorly understood. Here, we rigorously characterize the behavior of transformers with hardmax self-attention and normalization sublayers as the number of layers tends to infinity. By viewing such transformers as discrete-time dynamical systems describing the evolution of points in a Euclidean space, and thanks to a geometric interpretation of the self-attention mechanism based on hyperplane separation, we show that the transformer inputs asymptotically converge to a clustered equilibrium determined by special points called leaders. We then leverage this theoretical understanding to solve sentiment analysis problems from language processing using a fully interpretable transformer model, which effectively captures `context' by clustering meaningless words around leader words carrying the most meaning. Finally, we outline remaining challenges to bridge the gap between the mathematical analysis of transformers and their real-life implementation.

通过将transformers视为将导致点在欧几里得空间中演化的离散时间动态系统，并利用基于超平面分离的自注意力机制的几何解释，我们严格描述了层数趋近于无穷大时具有硬最大自注意和规范化子层的transformers行为。我们的理论理解使我们能够使用一个完全可解释的transformer模型来解决语言处理中的情感分析问题，这个模型通过将无意义的单词聚类到携带最有意义的主导单词周围来有效地捕捉`语境'。最后，我们概述了将transformers的数学分析与实际实现之间的差距的剩余挑战。

基于纯注意力硬极大值变压器的聚类及其在情感分析中的作用