Transformer, composed of self-attention and Feed-Forward Network, has
revolutionized the landscape of network design across various vision tasks. FFN
is a versatile operator seamlessly integrated into nearly all AI models to
effectively harness rich representations. Recent works also show that FFN
functions like key-value memories. Thus, akin to the query-key-value mechanism
within self-attention, FFN can be viewed as a memory network, where the input
serves as query and the two projection weights operate as keys and values,
respectively. We hypothesize that the importance lies in query-key-value
framework itself rather than in self-attention. To verify this, we propose
converting self-attention into a more FFN-like efficient token mixer with only
convolutions while retaining query-key-value framework, namely FFNification.
Specifically, FFNification replaces query-key and attention coefficient-value
interactions with large kernel convolutions and adopts GELU activation function
instead of softmax. The derived token mixer, FFNified attention, serves as
key-value memories for detecting locally distributed spatial patterns, and
operates in the opposite dimension to the ConvNeXt block within each
corresponding sub-operation of the query-key-value framework. Building upon the
above two modules, we present a family of Fast-Forward Networks. Our FFNet
achieves remarkable performance improvements over previous state-of-the-art
methods across a wide range of tasks. The strong and general performance of our
proposed method validates our hypothesis and leads us to introduce MetaMixer, a
general mixer architecture that does not specify sub-operations within the
query-key-value framework. We show that using only simple operations like
convolution and GELU in the MetaMixer can achieve superior performance.

通过将自注意力转换为采用大型卷积和 GELU 激活函数的更类似于 Feed-Forward Network 的有效令牌混合器，作者尝试验证了查询 - 键 - 值框架的重要性，并提出了一种新的混合器体系结构 MetaMixer，在仅使用简单操作（如卷积和 GELU）的情况下取得了卓越的性能改进。