Multi-head-self-attention (MHSA)-equipped models have achieved notable
performance in computer vision. Their computational complexity is proportional
to quadratic numbers of pixels in input feature maps, resulting in slow
processing, especially when dealing with high-resolution images. New types of
token-mixer are proposed as an alternative to MHSA to circumvent this problem:
an FFT-based token-mixer, similar to MHSA in global operation but with lower
computational complexity. However, despite its attractive properties, the
FFT-based token-mixer has not been carefully examined in terms of its
compatibility with the rapidly evolving MetaFormer architecture. Here, we
propose a novel token-mixer called dynamic filter and DFFormer and CDFFormer,
image recognition models using dynamic filters to close the gaps above.
CDFFormer achieved a Top-1 accuracy of 85.0%, close to the hybrid architecture
with convolution and MHSA. Other wide-ranging experiments and analysis,
including object detection and semantic segmentation, demonstrate that they are
competitive with state-of-the-art architectures; Their throughput and memory
efficiency when dealing with high-resolution image recognition is convolution
and MHSA, not much different from ConvFormer, and far superior to CAFormer. Our
results indicate that the dynamic filter is one of the token-mixer options that
should be seriously considered. The code is available at
this https URL

提出了一种新的动态滤波器 Token-mixer，与 MHSA 类似，但计算复杂度较低，并在图像识别中达到了与卷积和 MHSA 混合架构相近的精度水平，对于处理高分辨率图像的吞吐量和内存效率优于 CAFormer。