Transformers have established themselves as the leading neural network model
in natural language processing and are increasingly foundational in various
domains. In vision, the MLP-Mixer model has demonstrated competitive
performance, suggesting that attention mechanisms might not be indispensable.
Inspired by this, recent research has explored replacing attention modules with
other mechanisms, including those described by MetaFormers. However, the
theoretical framework for these models remains underdeveloped. This paper
proposes a novel perspective by integrating Krotov's hierarchical associative
memory with MetaFormers, enabling a comprehensive representation of the entire
Transformer block, encompassing token-/channel-mixing modules, layer
normalization, and skip connections, as a single Hopfield network. This
approach yields a parallelized MLP-Mixer derived from a three-layer Hopfield
network, which naturally incorporates symmetric token-/channel-mixing modules
and layer normalization. Empirical studies reveal that symmetric interaction
matrices in the model hinder performance in image recognition tasks.
Introducing symmetry-breaking effects transitions the performance of the
symmetric parallelized MLP-Mixer to that of the vanilla MLP-Mixer. This
indicates that during standard training, weight matrices of the vanilla
MLP-Mixer spontaneously acquire a symmetry-breaking configuration, enhancing
their effectiveness. These findings offer insights into the intrinsic
properties of Transformers and MLP-Mixers and their theoretical underpinnings,
providing a robust framework for future model design and optimization.

通过将 Krotov 的分层关联记忆与 MetaFormers 相结合，该论文提出了一种新的视角，将整个 Transformer 块的完整表示，包括标记 -/ 通道混合模块、层归一化和跳跃连接，作为一个单一的 Hopfield 网络。该方法产生了一个并行化的从三层 Hopfield 网络推导出的 MLP-Mixer，自然地融合了对称的标记 -/ 通道混合模块和层归一化。实证研究揭示了模型中的对称交互矩阵阻碍了图像识别任务的性能。引入破坏对称效果将对称并行化的 MLP-Mixer 的性能过渡到普通 MLP-Mixer 的性能。这表明在标准训练过程中，普通 MLP-Mixer 的权重矩阵自发地获得对称破缺配置，增强了其有效性。这些发现为 Transformer 和 MLP-Mixer 的内在特性及其理论基础提供了见解，为未来模型设计和优化提供了一个稳健的框架。

分层联想记忆、并行化 MLP-Mixer 和对称性破坏

Hierarchical Associative Memory, Parallelized MLP-Mixer, and Symmetry  Breaking

Spatio-temporal predictive learning is a learning paradigm that enables
models to learn spatial and temporal patterns by predicting future frames from
given past frames in an unsupervised manner. Despite remarkable progress in
recent years, a lack of systematic understanding persists due to the diverse
settings, complex implementation, and difficult reproducibility. Without
standardization, comparisons can be unfair and insights inconclusive. To
address this dilemma, we propose OpenSTL, a comprehensive benchmark for
spatio-temporal predictive learning that categorizes prevalent approaches into
recurrent-based and recurrent-free models. OpenSTL provides a modular and
extensible framework implementing various state-of-the-art methods. We conduct
standard evaluations on datasets across various domains, including synthetic
moving object trajectory, human motion, driving scenes, traffic flow and
weather forecasting. Based on our observations, we provide a detailed analysis
of how model architecture and dataset properties affect spatio-temporal
predictive learning performance. Surprisingly, we find that recurrent-free
models achieve a good balance between efficiency and performance than recurrent
models. Thus, we further extend the common MetaFormers to boost recurrent-free
spatial-temporal predictive learning. We open-source the code and models at
this https URL

本文提出了 OpenSTL，这是深度学习领域中，一个比较全面的基准测试，用于对空间 - 时间预测学习方法进行分类和评估，鲜见较为系统的研究，论文还对时序模型和数据集因素进行了详细分析，并提出了一种可提高空间 - 时间预测学习性能的通用 MetaFormers 模型。