In recent years, Vision Transformers have attracted increasing interest from
computer vision researchers. However, the advantage of these transformers over
CNNs is only fully manifested when trained over a large dataset, mainly due to
the reduced inductive bias towards spatial locality within the transformer's
self-attention mechanism. In this work, we present a data-efficient vision
transformer that does not rely on self-attention. Instead, it employs a novel
generalization to multiple axes of the very recent Hyena layer. We propose
several alternative approaches for obtaining this generalization and delve into
their unique distinctions and considerations from both empirical and
theoretical perspectives.
Our empirical findings indicate that the proposed Hyena N-D layer boosts the
performance of various Vision Transformer architectures, such as ViT, Swin, and
DeiT across multiple datasets. Furthermore, in the small dataset regime, our
Hyena-based ViT is favorable to ViT variants from the recent literature that
are specifically designed for solving the same challenge, i.e., working with
small datasets or incorporating image-specific inductive bias into the
self-attention mechanism. Finally, we show that a hybrid approach that is based
on Hyena N-D for the first layers in ViT, followed by layers that incorporate
conventional attention, consistently boosts the performance of various vision
transformer architectures.

通过使用基于多个轴的新型 Hyena 层，我们提出了一种不依赖于自注意力机制的高效视觉 Transformer，并且实验证明该方法在多个数据集上提升了各种 Vision Transformer 架构的性能。