Transformers are extremely successful machine learning models whose
mathematical properties remain poorly understood. Here, we rigorously
characterize the behavior of transformers with hardmax self-attention and
normalization sublayers as the number of layers tends to infinity. By viewing
such transformers as discrete-time dynamical systems describing the evolution
of points in a Euclidean space, and thanks to a geometric interpretation of the
self-attention mechanism based on hyperplane separation, we show that the
transformer inputs asymptotically converge to a clustered equilibrium
determined by special points called leaders. We then leverage this theoretical
understanding to solve sentiment analysis problems from language processing
using a fully interpretable transformer model, which effectively captures
`context' by clustering meaningless words around leader words carrying the most
meaning. Finally, we outline remaining challenges to bridge the gap between the
mathematical analysis of transformers and their real-life implementation.

通过将 transformers 视为将导致点在欧几里得空间中演化的离散时间动态系统，并利用基于超平面分离的自注意力机制的几何解释，我们严格描述了层数趋近于无穷大时具有硬最大自注意和规范化子层的 transformers 行为。我们的理论理解使我们能够使用一个完全可解释的 transformer 模型来解决语言处理中的情感分析问题，这个模型通过将无意义的单词聚类到携带最有意义的主导单词周围来有效地捕捉 ` 语境 '。最后，我们概述了将 transformers 的数学分析与实际实现之间的差距的剩余挑战。

基于纯注意力硬极大值变压器的聚类及其在情感分析中的作用

Clustering in pure-attention hardmax transformers and its role in  sentiment analysis

Interpretable mathematical expressions defining discrete-time dynamical
systems (iterated maps) can model many phenomena of scientific interest,
enabling a deeper understanding of system behaviors. Since formulating
governing expressions from first principles can be difficult, it is of
particular interest to identify expressions for iterated maps given only their
data streams. In this work, we consider a modified Symbolic Artificial Neural
Network-Trained Expressions (SymANNTEx) architecture for this task, an
architecture more expressive than others in the literature. We make a
modification to the model pipeline to optimize the regression, then
characterize the behavior of the adjusted model in identifying several
classical chaotic maps. With the goal of parsimony, sparsity-inducing weight
regularization and information theory-informed simplification are implemented.
We show that our modified SymANNTEx model properly identifies single-state maps
and achieves moderate success in approximating a dual-state attractor. These
performances offer significant promise for data-driven scientific discovery and
interpretation.

通过对修正的 SymANNTEx 模型进行权重正则化和信息论简化，本研究在识别经典混沌映射中成功识别单状态映射，并在逼近双状态吸引子方面取得了适度的成果，这为基于数据的科学发现和解释提供了重要的前景。