Vision-Transformers are widely used in various vision tasks. Meanwhile, there is another line of works starting with the MLP-mixer trying to achieve similar performance using mlp-based architectures. Interestingly, until now none reported using them for NLP tasks, additionally until now non of those mlp-based architectures claimed to achieve state-of-the-art in vision tasks. In this paper, we analyze the expressive power of mlp-based architectures in modeling dependencies between multiple different inputs simultaneously, and show an exponential gap between the attention and the mlp-based mechanisms. Our results suggest a theoretical explanation for the mlp inability to compete with attention-based mechanisms in NLP problems, they also suggest that the performance gap in vision tasks may be due to the mlp relative weakness in modeling dependencies between multiple different locations, and that combining smart input permutations to the mlp architectures may not suffice alone to close the performance gap.

本文分析了基于MLP的体系结构模拟多个不同输入之间依赖关系的表达能力，并展示了注意力机制和基于MLP的机制之间的指数差距，从而提供了MLP无法与注意力机制在NLP问题中竞争的理论解释。此外，文中还提出，MLP相对于注意力机制在建模多个不同位置之间的依赖关系的能力较弱，而将智能输入排列与MLP体系结构结合可能不足以单独消除性能差距。

Transformer vs. MLP-Mixer: 自然语言处理问题的指数表达差距