Prior works have proposed several strategies to reduce the computational cost
of self-attention mechanism. Many of these works consider decomposing the
self-attention procedure into regional and local feature extraction procedures
that each incurs a much smaller computational complexity. However, regional
information is typically only achieved at the expense of undesirable
information lost owing to down-sampling. In this paper, we propose a novel
Transformer architecture that aims to mitigate the cost issue, named Dual
Vision Transformer (Dual-ViT). The new architecture incorporates a critical
semantic pathway that can more efficiently compress token vectors into global
semantics with reduced order of complexity. Such compressed global semantics
then serve as useful prior information in learning finer pixel level details,
through another constructed pixel pathway. The semantic pathway and pixel
pathway are then integrated together and are jointly trained, spreading the
enhanced self-attention information in parallel through both of the pathways.
Dual-ViT is henceforth able to reduce the computational complexity without
compromising much accuracy. We empirically demonstrate that Dual-ViT provides
superior accuracy than SOTA Transformer architectures with reduced training
complexity. Source code is available at
https://github.com/YehLi/ImageNetModel.

本文提出了名为 Dual Vision Transformer (Dual-ViT) 的新型 Transformer 结构，在保持准确度的前提下，通过综合运用语义路径和像素路径，将全局语义压缩成更高效的先验信息，进而实现了减少计算复杂度的目的，并在实验中表现出比 SOTA Transformer 结构更高的准确度。