In recent years, Vision Transformers have attracted increasing interest from
computer vision researchers. However, the advantage of these transformers over
CNNs is only fully manifested when trained over a large dataset, mainly due to
the reduced inductive bias towards spatial locality within the transformer's
self-attention mechanism. In this work, we present a data-efficient vision
transformer that does not rely on self-attention. Instead, it employs a novel
generalization to multiple axes of the very recent Hyena layer. We propose
several alternative approaches for obtaining this generalization and delve into
their unique distinctions and considerations from both empirical and
theoretical perspectives.
Our empirical findings indicate that the proposed Hyena N-D layer boosts the
performance of various Vision Transformer architectures, such as ViT, Swin, and
DeiT across multiple datasets. Furthermore, in the small dataset regime, our
Hyena-based ViT is favorable to ViT variants from the recent literature that
are specifically designed for solving the same challenge, i.e., working with
small datasets or incorporating image-specific inductive bias into the
self-attention mechanism. Finally, we show that a hybrid approach that is based
on Hyena N-D for the first layers in ViT, followed by layers that incorporate
conventional attention, consistently boosts the performance of various vision
transformer architectures.

通过使用基于多个轴的新型 Hyena 层，我们提出了一种不依赖于自注意力机制的高效视觉 Transformer，并且实验证明该方法在多个数据集上提升了各种 Vision Transformer 架构的性能。

多维鬣狗在空间归纳偏差中的应用

Multi-Dimensional Hyena for Spatial Inductive Bias

We investigate the connections between neural networks and simple building
blocks in kernel space. In particular, using well established feature space
tools such as direct sum, averaging, and moment lifting, we present an algebra
for creating "compositional" kernels from bags of features. We show that these
operations correspond to many of the building blocks of "neural tangent kernels
(NTK)". Experimentally, we show that there is a correlation in test error
between neural network architectures and the associated kernels. We construct a
simple neural network architecture using only 3x3 convolutions, 2x2 average
pooling, ReLU, and optimized with SGD and MSE loss that achieves 96% accuracy
on CIFAR10, and whose corresponding compositional kernel achieves 90% accuracy.
We also use our constructions to investigate the relative performance of neural
networks, NTKs, and compositional kernels in the small dataset regime. In
particular, we find that compositional kernels outperform NTKs and neural
networks outperform both kernel methods.

通过研究神经网络和内核空间中的简单构建块之间的联系，我们提出了一种从特征组中创建 “组合” 内核的代数。在实验中，我们发现神经网络体系结构和相关内核的测试误差之间存在相关性，并构建了一个只使用 3x3 卷积、2x2 平均池化、ReLU 并使用 SGD 和 MSE 损失进行优化的简单神经网络体系结构，在 CIFAR10 数据集上获得 96％的准确度，其对应的组合内核可达到 90％的准确度。我们还使用我们的构造方法研究了神经网络、NTK 和组合内核在小数据集环境中的相对性能表现，特别是发现组合内核优于 NTK，神经网络优于这两种内核方法。