Transformer-based models have demonstrated exceptional performance across
diverse domains, becoming the state-of-the-art solution for addressing
sequential machine learning problems. Even though we have a general
understanding of the fundamental components in the transformer architecture,
little is known about how they operate or what are their expected dynamics.
Recently, there has been an increasing interest in exploring the relationship
between attention mechanisms and Hopfield networks, promising to shed light on
the statistical physics of transformer networks. However, to date, the
dynamical regimes of transformer-like models have not been studied in depth. In
this paper, we address this gap by using methods for the study of asymmetric
Hopfield networks in nonequilibrium regimes --namely path integral methods over
generating functionals, yielding dynamics governed by concurrent mean-field
variables. Assuming 1-bit tokens and weights, we derive analytical
approximations for the behavior of large self-attention neural networks coupled
to a softmax output, which become exact in the large limit size. Our findings
reveal nontrivial dynamical phenomena, including nonequilibrium phase
transitions associated with chaotic bifurcations, even for very simple
configurations with a few encoded features and a very short context window.
Finally, we discuss the potential of our analytic approach to improve our
understanding of the inner workings of transformer models, potentially reducing
computational training costs and enhancing model interpretability.

使用非平衡 Hopfield 网络的路径积分方法研究了变压器网络的动力学规律，发现了与混沌分叉相关的非平衡相变等非平凡的动力学现象，并讨论了这种分析方法改善对变压器模型内部运作理解的潜力。