In this paper, we observe two levels of redundancies when applying vision
transformers (ViT) for image recognition. First, fixing the number of tokens
through the whole network produces redundant features at the spatial level.
Second, the attention maps among different transformer layers are redundant.
Based on the observations above, we propose a PSViT: a ViT with token Pooling
and attention Sharing to reduce the redundancy, effectively enhancing the
feature representation ability, and achieving a better speed-accuracy
trade-off. Specifically, in our PSViT, token pooling can be defined as the
operation that decreases the number of tokens at the spatial level. Besides,
attention sharing will be built between the neighboring transformer layers for
reusing the attention maps having a strong correlation among adjacent layers.
Then, a compact set of the possible combinations for different token pooling
and attention sharing mechanisms are constructed. Based on the proposed compact
set, the number of tokens in each layer and the choices of layers sharing
attention can be treated as hyper-parameters that are learned from data
automatically. Experimental results show that the proposed scheme can achieve
up to 6.6% accuracy improvement in ImageNet classification compared with the
DeiT.

本篇论文提出了一种 PSViT 视觉变换器，通过令每个 transformer 层与其相邻层共享注意力映射和减少空间级别上的特征数量来降低冗余，实现更好的速度和准确性