Considering the spectral properties of images, we propose a new
self-attention mechanism with highly reduced computational complexity, up to a
linear rate. To better preserve edges while promoting similarity within
objects, we propose individualized processes over different frequency bands. In
particular, we study a case where the process is merely over low-frequency
components. By ablation study, we show that low frequency self-attention can
achieve very close or better performance relative to full frequency even
without retraining the network. Accordingly, we design and embed novel
plug-and-play modules to the head of a CNN network that we refer to as FsaNet.
The frequency self-attention 1) takes low frequency coefficients as input, 2)
can be mathematically equivalent to spatial domain self-attention with linear
structures, 3) simplifies token mapping ($1\times1$ convolution) stage and
token mixing stage simultaneously. We show that the frequency self-attention
requires $87.29\% \sim 90.04\%$ less memory, $96.13\% \sim 98.07\%$ less FLOPs,
and $97.56\% \sim 98.18\%$ in run time than the regular self-attention.
Compared to other ResNet101-based self-attention networks, FsaNet achieves a
new state-of-the-art result ($83.0\%$ mIoU) on Cityscape test dataset and
competitive results on ADE20k and VOCaug.

通过对图像的频谱特性的研究，我们提出了一个高度降低计算复杂度的自注意机制。通过在不同频段上的个性化处理，可以更好地保留边缘，并在对象内提高相似性。我们设计并嵌入新的插入式模块到 CNN 网络的头部，称为 FsaNet。通过低频自注意力的应用，相对于全频，我们显示可以实现非常接近或更好的性能。相比其他基于 ResNet101 的自注意网络，FsaNet 在 Cityscape 测试数据集上取得了新的最优结果（83.0％mIoU），在 ADE20k 和 VOCaug 上取得了竞争性的结果。